Home › Research › Local and service businesses
SEMPITE Research · Measurement Study · September 2026
We Crawled 1,416 Real Estate Agent Websites. Only 1 in 5 Is Ready for AI.
In Part 1 we found the whole industry is guessing at what AI can and can’t see. So we measured it. We harvested 1,416 independent agent and brokerage sites across ~55 US markets and read each one the way a non-rendering AI crawler does. Most render fine for a human and show an AI assistant no listings, no reviews, and barely an identity.
What we did
We built a sampling frame of 1,416 independent real estate agent, team, and local-brokerage websites by harvesting search results across roughly 55 US markets in every region — from New York, LA and Chicago down to Boise, Fargo, Honolulu and Anchorage — with the national portals (Zillow, Realtor.com, Redfin, Homes.com, Trulia) and lead-gen aggregators deliberately excluded. Then we crawled each one the way an AI answer engine does: static HTML, no JavaScript executed, exactly the posture of GPTBot, ClaudeBot, PerplexityBot and CCBot.
1,097 (77.5%) were reachable on a static fetch. Every metric below is computed on those 1,097 — the sites an AI crawler can actually retrieve — because that is the fair denominator.
The site looks fine. The listings aren’t there.
We found a listings or property-search page on 59.3% of reachable sites. Of those 650 sites, 30.5% (95% CI 27.0–34.1) exposed no machine-readable property data at all — no price, no beds, no baths, no square footage, no MLS number — in the HTML their server sends.
These pages aren’t blank — a median of ~2,600 characters of navigation and marketing text is right there. What’s missing is the homes. The listings are injected by a JavaScript IDX/MLS widget, and the crawlers that feed AI answers don’t run JavaScript. A human sees a grid of properties; the AI sees the page furniture and no inventory.
Almost no machine-readable identity, and zero listing schema
Structured data is how an answer engine knows who an agent is and what they have. Real estate barely uses it. Here is what the 1,097 reachable sites declare in server-side JSON-LD:
Most of the “any schema” is generic Organization or BreadcrumbList boilerplate from a website template. Only 29.4% declare themselves a RealEstateAgent, and not one of the 1,097 sites published RealEstateListing schema for its inventory. The single most relevant schema type for the industry is used by nobody.
95% give the AI no reason to trust them
Reviews are the strongest signal a local agent can send an answer engine — and the one place an independent can out-signal a national portal. Yet 95.4% of reachable sites (only 4.6%, 95% CI 3.5–6.0, expose any) publish no AggregateRating or Review data a machine can read. The five-star widgets are on the page for humans, rendered in JavaScript, invisible to the system doing the recommending.
1 in 9 blocks an AI crawler at the door — probably by accident
10.8% of sites (95% CI 9.0–12.8) disallow at least one core AI crawler in robots.txt. The tell that it isn’t deliberate: the block rate is nearly identical across bots that have nothing to do with each other.
A blanket Disallow copied from a template quietly opts an agent out of ChatGPT, Claude, Perplexity, Amazon’s Rufus and Google’s AI in one line.
Half are on an IDX platform — and that’s where the invisibility comes from
48.1% of reachable sites run on a detectable IDX or website platform. Among those with a listings page, 27.3% serve no property data in HTML — the JavaScript-widget pattern, baked in by the platform.
Showcase IDX (4.0%), kvCORE (3.7%), Wix (2.6%), Placester (1.8%), iHomefinder (1.7%) and BoomTown (1.0%) follow. The choice of platform is quietly deciding whether an agent’s listings exist to AI at all.
Put together: only 1 in 5 is AI-ready
Combine the gates — allows the core AI crawlers and serves real listing data in HTML and declares an agent or business entity — and just 19.9% of reachable sites clear the bar (95% CI 17.6–22.3). The good news buried in the numbers: the fixes are unglamorous and cheap. Server-side render the listing and location pages, add RealEstateAgent and Review schema, delete the stray robots block. None of it requires out-spending a portal — it requires being readable, which almost nobody is.
Frequently asked questions
How were the 1,416 sites chosen?
Search-harvested across ~55 US markets in every region (incl. Hawaii and Alaska), with national portals and lead-gen aggregators excluded. 1,097 (77.5%) were reachable and all metrics are computed on those. Franchise-affiliated agents with their own branded domain are included, since that domain is what an AI crawler reads.
What does “AI-ready” mean?
A composite: allows the core AI crawlers in robots.txt, exposes real listing data (price/beds/baths/sq ft/MLS) in server-rendered HTML, and declares a RealEstateAgent or LocalBusiness entity. Only 19.9% cleared all three.
Why can’t AI see the listings on 30% of sites?
Of the 650 sites with a listings page, 30.5% served no property data in HTML. The pages have ~2,600 median chars of other text but the listings are injected by a JavaScript IDX widget, and AI crawlers don’t run JavaScript.
What’s the biggest gap?
Structured data and reviews. 0 of 1,097 sites use RealEstateListing schema and only 29.4% declare a RealEstateAgent; 95.4% expose no machine-readable reviews.
Are agents blocking AI on purpose?
Mostly not. 10.8% block a core AI crawler, with near-identical rates across unrelated bots (Amazonbot 11.4%, GPTBot 10.6%), which points to copied blanket robots rules.
Method, in short
Sampling frame of 1,416 independent agent/brokerage/team domains harvested via search across ~55 US markets (portals and aggregators excluded), crawled 16 September 2026. For each site we fetched robots.txt and parsed the AI-bot policy; fetched the homepage and extracted JSON-LD schema, review markup, and contact exposure; discovered a listings/search page from homepage links and fetched it; and scanned the served HTML for machine-readable listing signals (price, beds, baths, sq ft, MLS #). All fetches were static, no JavaScript executed, mirroring the non-rendering AI crawlers. Proportions carry Wilson score intervals and are computed on the 1,097 reachable sites.
Known limits, stated plainly: a static fetch cannot see JavaScript-rendered content — which is the point, because non-rendering AI crawlers can’t either, though some agents (Googlebot, Gemini, AppleBot) do render and would see more than we credit here; we measure one listings page per site, discovered from homepage links, so sites that hide their search behind a form or an unlabeled link may be undercounted as “no listings page found”; the frame is search-harvested rather than a census, so it skews toward agents with enough visibility to surface in results; IDX-platform detection is fingerprint-based and conservative; and 22.5% of the frame was unreachable on a static fetch (190 returned a bot-block status), which is itself a mild readiness signal we did not count against the metrics. Raw per-site data and the crawler ship with the study.
We run this readiness check against an individual agent or brokerage site, then fix what an AI answer engine can’t read.
AI Search Visibility (GEO)