We checked 740 of the world's top websites for AI bots. Most that block GPTBot also block ChatGPT search

Originally published on pagelens.io.

Most advice about AI search starts with "check your robots.txt". I wanted to know what the biggest sites actually do, so I ran an AI visibility check on the top 1,500 domains of the Tranco list (list 4672X), a research ranking of the most visited domains, on October 12, 2026.

Many of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). 740 served a home page to a normal browser request, and every number below is out of those 740 websites.

The short version

• 15% block GPTBot in robots.txt, and about as many block ClaudeBot (14%) and Google-Extended (13%).
• 13% block at least one agent that answers people: OAI-SearchBot, ChatGPT-User, Perplexity's bots or Claude-User.
• Of the 109 sites that block GPTBot, 69 (63%) also block ChatGPT's search or browsing agent. Only 40 block training and keep ChatGPT search open.
• 9% turn away requests that carry an AI bot's name at the server, even though their robots.txt allows those bots.
• About 14% publish an llms.txt; 45% have structured data on the home page; 38% have no sitemap I could find.

1. robots.txt: training bots are blocked more than answer bots

Share of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):

Most of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.

2. Blocking GPTBot usually blocks ChatGPT search too

GPTBot only collects training data. Whether a site appears in ChatGPT search depends on OAI-SearchBot, and whether ChatGPT can open a page while answering someone depends on ChatGPT-User. OpenAI documents them as separate agents with separate robots.txt rules.

Of the 109 websites that block GPTBot:

• 69 also block OAI-SearchBot or ChatGPT-User. Among them are cnn.com, bbc.com, reddit.com, imdb.com, espn.com, nature.com and Amazon's stores. They opt out of ChatGPT as a whole.
• 40 block only training and leave ChatGPT search and browsing open, including forbes.com, bloomberg.com, airbnb.com, figma.com, ted.com and nintendo.com.

Both are legitimate choices. The second one is what most businesses want: stay out of training, stay in the answers. If you copied a "block AI bots" robots.txt from a blog post, check which group you're in.

3. robots.txt isn't the whole story

For each site I also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot) and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot.

I then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:

• 69 (9.3% of all sites) let the browser and the generic crawler through, but turn away requests named after AI bots. 37 of them turn away ChatGPT-User, the agent that opens pages to answer people.
• 37 (5%) turn away any bot-like request, AI or not.
• 19 did not repeat on the second test, and 5 blocked the browser too.

One caution: these requests don't come from OpenAI's or Anthropic's servers. Some CDNs block requests that claim to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, "blocks AI bots" and "blocks impostors" look the same, which is why I don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.

The useful lesson for a site owner: robots.txt can say "allowed" while the server says no, and nothing in robots.txt will show it.

4. Can AI read the page once it gets it?

AI crawlers don't run JavaScript. On 22% of home pages, fewer than 120 words were readable in the HTML before scripts run. This is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.

llms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare "OK", an error page), which a naive check would count.

What to do on your own site

1. Decide training and answers separately. Blocking GPTBot and ClaudeBot keeps you out of training; to stay in ChatGPT and Claude answers, allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User.
2. Check the server, not only the file. CDN bot settings and firewalls override robots.txt.
3. Make sure the text is in the HTML. Server rendering or prerendering for the pages you want quoted.
4. Then measure what AI actually does. Server logs show which pages AI crawlers read and which ones assistants open to answer people.

If you want to run the same check on your site, I built it as a free tool (no sign-up): pagelens.io/tools/ai-crawler-checker.

Method and limits

• Domains: top 1,500 of Tranco list 4672X. A domain counts as a website if a normal browser request to its home page got a page (2xx or 3xx, no challenge). 740 did.
• robots.txt rules were evaluated for the home page path with the most specific matching user-agent group, as crawlers do.
• Bot requests used each bot's name and said they came from the checker. Run from Central Asia on October 12, 2026; results can differ by location and change at any time.
• These are shares, not judgements: blocking AI bots is a valid choice.

Happy to answer questions about the data or the method in the comments.