Block AI Crawlers? GPTBot vs ChatGPT-User

Why user-agent blocking mostly punishes the polite bots

A robots.txt rule only stops crawlers that read robots.txt and choose to obey it. OpenAI, Anthropic, Google, and Perplexity publish their user agents and honor disallow rules. The scrapers you actually worry about do not announce themselves and do not read the file. So a blanket block on AI user agents keeps out the companies that asked permission and does nothing to the ones that did not. If you want real enforcement you need it at the edge, through your CDN or firewall, not in a text file.

What Cloudflare pay-per-crawl actually is and why adoption is tiny

Cloudflare’s pay-per-crawl lets a site return a payment-required response to AI crawlers and name a price. It is a sensible idea for the handful of sites whose content has standalone training value. For everyone else the economics do not work, because the marginal training value of a single small site is close to zero. The licensing money in this market goes to bulk corpora, Reddit-scale archives and large publishers, negotiated as deals rather than metered per page. A local dentist or a boutique consultancy setting a crawl price is not going to get paid. It is going to get skipped.

Two kinds of sites, two right answers

If your site is the product, meaning a publisher, a paid course, a proprietary dataset, or original reporting that people pay to read, blocking training crawlers is defensible. The model gains what you sell, and you gain nothing back. Block the training agents, keep the user-triggered ones, and consider the edge controls above.

Plenty of sites are both. A consultancy with a paid course, a publication with a services page, a software company with documentation it would rather not train a competitor on. The answer there is not all-or-nothing. Disallow the training agents from the folder that holds the paid or proprietary material and leave the rest of the site open. Robots.txt works per path, and the user-triggered agents can still reach the pages that exist to bring customers in.

If your site exists so that people can find your business, blocking is self-harm. You built the site to be discovered. AI answers are now a discovery channel. Cutting the crawlers off is the same decision as removing yourself from Google in 2010 because Google was profiting from your listing. Technically true, commercially backwards.

What 307 SEO agencies chose

We wanted a baseline from the people who advise everyone else, so we took the top 20 organic results for 36 SEO-agency queries, including “seo agency” plus a city for 30 large US metros, cleaned out directories and job boards, and scored the 307 agency sites that remained. Only 3.6 percent block GPTBot. Under one percent block ChatGPT-User or PerplexityBot. Ninety-six percent allow OpenAI’s training crawler outright. Whatever agencies say in blog posts, on their own sites they have decided that being found by AI matters more than keeping a model from reading their service pages. The full rubric and distribution are in the agency AI-readiness census.

GPTBot is training, ChatGPT-User is a customer mid-question

OpenAI runs separate agents for separate jobs. GPTBot crawls the open web to build training data. ChatGPT-User fetches a specific page because a person in a ChatGPT conversation asked something that needs it, right now. OAI-SearchBot indexes for ChatGPT search. Perplexity and Anthropic make similar splits. That distinction is the whole policy. If you want to keep your content out of training but still be there when a customer asks, robots.txt can say exactly that:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

Adjust the training lines to taste. The point is that the user-triggered agents stay open. Our AI Readiness Score reads your robots.txt and flags accidental blocks on the agents that matter, and the llms.txt generator gives the ones you allow a clean map of what to read first.

Blocking your own site does not remove you from AI answers

This surprises people. In our category study of Google AI Overview citations, brand-owned websites received about five percent of citation slots. The other 95 percent cited third parties: reviews, directories, publications, forums, video. The engines will keep describing your business from those sources whether or not your own site is readable. Blocking does not give you control. It removes the one source you write yourself from the mix and leaves the rest. If a wrong description is already circulating, the fix is the one in why businesses disappear from AI search, not a disallow line.

The EU opt-out question

Google has proposed a mechanism that would let publishers opt out of having their content used in AI search features, and EU regulators have been gathering publisher views on it. It is worth watching, but the same logic applies. A publisher whose content is the product may want the opt-out. A business whose site exists to be found should be careful about opting out of the surface where customers are increasingly getting their answers. Perplexity’s handling of crawled content, covered in how Perplexity processes your content, shows how differently the engines behave once you let them in.