SEMPITE Research · Census · August 2026 · By Jacob Gerrish
We Scored 307 SEO Agencies With Our Own AI-Readiness Rubric
These are the agencies winning Google’s organic results for “SEO agency” across 30 US metros, the best-case sample. We ran every one of their websites through the same public, deterministic 24-check AI-readiness rubric we score client sites with. The median is respectable. The failure rates on individual checks are not.
What SEO agencies fail on their own sites
Share of the 307 agencies failing each check outright (partial credit not counted as failure). Single crawl, August 30, 2026.
All 24 checks, by failure rate
The chart above shows the ten failures that matter most. Here is the full rubric, grouped the way the scorer groups it, with the share of the 307 agencies that failed each check outright.
| Check | Area | Failed |
|---|---|---|
| sameAs profile links | Structured data | 37.1% |
| Content schema (FAQ/Article/Breadcrumb) | Structured data | 36.8% |
| Entity schema (Organization/LocalBusiness/Person) | Structured data | 29.0% |
| WebSite schema | Structured data | 28.3% |
| JSON-LD structured data present | Structured data | 18.2% |
| Google-Extended allowed | AI-crawler access | 4.6% |
| GPTBot allowed | AI-crawler access | 3.6% |
| ClaudeBot allowed | AI-crawler access | 3.3% |
| ChatGPT-User allowed | AI-crawler access | 0.3% |
| PerplexityBot allowed | AI-crawler access | 0.3% |
| H1 heading | Machine readability | 20.2% |
| Meta description | Machine readability | 16.6% |
| Canonical URL | Machine readability | 15.6% |
| Readable text content in HTML (not JS-only) | Machine readability | 13.7% |
| Page loads (HTTP 200, under 3s) | Machine readability | 11.4% |
| Title tag | Machine readability | 6.5% |
| llms.txt links to key pages | llms.txt | 54.4% |
| llms.txt file exists | llms.txt | 51.5% |
| FAQ / question-and-answer content | Answerability | 54.1% |
| Structured lists/sections | Answerability | 21.2% |
| Open Graph tags | Answerability | 17.6% |
| sitemap.xml | Discoverability | 22.8% |
| Favicon | Discoverability | 14.3% |
| robots.txt | Discoverability | 10.4% |
Read it by area and a pattern shows up. The classic machine-readability checks (title, H1, canonical, readable HTML) fail for 6% to 20% of agencies. The newer layers fail far more often: llms.txt for about half, structured data for up to 37%, and question-and-answer content for 54%. AI-crawler access is the exception, with every crawler allowed by at least 95% of sites.
The score distribution is bimodal
Most agencies cluster at 70–100. Then there is a second spike at 20–29. These are sites the rubric could barely read because of JavaScript-only rendering, bot walls, or broken responses. To a search crawler or an AI engine, those sites look the same way they looked to us.
What the census shows
1. The industry passes its own exam, with a gentleman’s C on the new material. The classic hygiene is mostly there: titles (93.5%), robots.txt (89.6%), fast-enough homepages (88.6%). The AI-era layer is where it thins out: half the industry has no llms.txt and no answer-formatted content, and more than a third has no content schema.
2. One in five has no structured data at all. Not imperfect markup, zero JSON-LD on the homepage of a business that sells search optimization. Structured data is how AI engines confirm who a business is; these agencies are asking clients to buy a discipline they haven’t applied to themselves.
3. One in ten ranks despite a site a machine can barely read. The 20–29 spike is 10% of the sample. They rank on reputation, links, and history. That proves authority can carry a weak site, and also that their example is a poor template for a small business that has no authority yet.
4. Agencies are not blocking AI. The paranoia narrative is wrong here. Only 3.6% block GPTBot, and under 1% block ChatGPT-User or PerplexityBot. Whatever agencies say about AI scraping, their robots.txt files have voted: they want to be in the answers.
The buyer’s takeaway: before hiring an SEO agency, spend sixty seconds checking whether they practice what they will bill you for. View their homepage source and look for application/ld+json; try their-domain.com/llms.txt, or run their domain through the same free rubric we used. It is public, deterministic, and takes ten seconds.
Method
Sample: the top 20 organic Google results (US) for 36 queries, six generic phrasings of “SEO agency/company” plus “seo agency {city}” for 30 large US metros, collected August 30, 2026. Directories, review platforms, job boards, media sites, and SaaS tools were removed by blocklist plus a manual pass (14 removals), leaving 307 unique agency domains. Each was scored once with SEMPITE’s public AI-readiness rubric: 24 deterministic checks across structured data, AI-crawler access in robots.txt, machine readability, llms.txt, answerability, and discoverability; the full rubric is published. No agency is named; this is a census, not a leaderboard.
Limitations
The rubric is ours, and it weights the AI-visibility layer we specialize in. We disclose that, and the scoring is deterministic and reproducible by anyone. Scores reflect the homepage and site-level files on one crawl date; JavaScript-only sites under-score by design, since AI engines reading raw HTML face the same wall. The sample is agencies that already rank: a best-case selection, which makes the failure rates conservative for the industry as a whole.
Frequently asked questions
How AI-ready are SEO agencies' own websites?
The median of 307 agencies scored 82 out of 100 on SEMPITE's 24-check AI-readiness rubric, but 13% scored under 50 and one in five has no JSON-LD structured data at all.
Which agencies were included?
The 307 agencies winning Google's organic results for SEO agency queries across 30 US metros, the best-case sample, crawled once on 30 August 2026.
What do agencies fail on most?
No FAQ or question-and-answer content at 54.1%, no llms.txt file at 51.5%, no sameAs profile links at 37.1%, no content schema at 36.8%, and no Organization schema at 29%.
Are SEO agencies blocking AI crawlers?
No. 96% allow GPTBot. Only 3.6% block it, and under 1% block ChatGPT-User or PerplexityBot.
Why is the score distribution bimodal?
Most agencies cluster between 70 and 100, but 31 sites scored 20 to 29, about 10% of the sample: JavaScript-only rendering, bot walls or broken responses that a crawler or AI engine can barely read. They rank on reputation and links, not on their sites.
Which checks do SEO agencies pass most often?
AI-crawler access. ChatGPT-User and PerplexityBot are blocked by only 0.3% of agencies, ClaudeBot by 3.3%, GPTBot by 3.6% and Google-Extended by 4.6%. A title tag is missing on 6.5% of sites.
Would your site pass the same 24 checks?
Run the free rubric: the identical scoring used in this census, with your fixes listed.