How the top 2,000 websites treat AI crawlers: 2026 data
By Anshad P.V, Founder, Axpecto · · 4 min read
Of the world's most visited websites, about one in five blocks at least one AI crawler in robots.txt, and one in six blocks a crawler that powers a major AI assistant. We checked the top 2,000 domains on the Tranco ranking on 23 October 2026: their robots.txt rules for 16 AI and search crawlers, whether they publish an llms.txt and a sitemap, what structured data the homepage carries, and how much of the homepage a crawler that doesn't run JavaScript can read.
Every figure below comes from the same code as our free AI Crawler Checker, so you can run the identical check on your own site.
How did we run the study?
- Sample: the top 2,000 domains of the Tranco list (list ID 56Z8N), a research ranking that combines several traffic sources.
- Of those, 863 served a homepage to a plain request. The rest are infrastructure domains with no website (CDNs, DNS and API hosts), or refused automated requests. Every percentage below is out of those 863 sites, or out of the 697 whose robots.txt we could read.
- robots.txt was read by group, as the standard (RFC 9309) says: a group naming a crawler replaces the * group for it. We report whether each crawler may fetch the site root. A robots.txt that bot protection refused is counted as unknown, never as blocking.
- The homepage was fetched once without running JavaScript, the way most AI crawlers read a page.
- One run, on one day. Sites change their rules, so read these as a snapshot.
How many top websites block AI crawlers?
Out of 697 sites with a readable robots.txt, 147 (21.1%) block at least one AI crawler at the site root, and 114 (16.4%) block at least one of the five that matter most for AI assistants: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. 47 sites (6.7%) block all five.
| Crawler | Company | Blocked at the site root |
|---|---|---|
| Bytespider | ByteDance | 14.8% |
| GPTBot | OpenAI (training) | 13.5% |
| cohere-ai | Cohere | 11.9% |
| ClaudeBot | Anthropic (training) | 11.8% |
| Google-Extended | Google (Gemini use) | 11.6% |
| meta-externalagent | Meta | 11.3% |
| Applebot-Extended | Apple | 10.3% |
| Amazonbot | Amazon | 10.3% |
| anthropic-ai | Anthropic (older token) | 10.2% |
| PerplexityBot | Perplexity (search index) | 9.2% |
| ChatGPT-User | OpenAI (live fetch) | 7.9% |
| Perplexity-User | Perplexity (live fetch) | 7.3% |
| Claude-User | Anthropic (live fetch) | 6.5% |
| OAI-SearchBot | OpenAI (ChatGPT search) | 5.7% |
| Bingbot | Microsoft | 2.9% |
| Googlebot | 2.0% |
Most sites (76.8%) never name an AI crawler in robots.txt at all, so AI crawlers simply follow the same rules as everyone else.
Do sites block AI training or AI search?
Mostly training. GPTBot, which collects pages for OpenAI's training, is blocked more than twice as often as OAI-SearchBot, which builds the index ChatGPT search uses (13.5% against 5.7%). 54 sites (7.7%) block GPTBot while allowing OAI-SearchBot: they opt out of training and stay findable in ChatGPT search. The same pattern holds for Anthropic, where ClaudeBot (training, 11.8%) is blocked more than Claude-User (live fetch, 6.5%).
If you want to make the same choice, our guide to robots.txt for AI crawlers has the rules to copy.
How many top websites have an llms.txt?
146 of the 863 sites (16.9%) publish an llms.txt, the plain-language summary of a site written for AI assistants. A sitemap is far more common: 63.8% have one. We explain what goes in the file in What is llms.txt?
How many homepages use structured data?
| On the homepage | Share of sites |
|---|---|
| Any JSON-LD structured data | 48.3% |
| Organization (or a subtype) schema | 32.4% |
| Organization schema with sameAs profile links | 21.1% |
| A meta description | 80.4% |
Organization schema with sameAs links is how a site tells an assistant which company it is and which profiles confirm it. Fewer than one top site in four does it. See Structured data for AI search for the fields that matter.
How much does a crawler that doesn't run JavaScript see?
The median homepage served 817 words of text without JavaScript. But 146 sites (16.9%) served fewer than 50 words: their content appears only after JavaScript runs, or the plain request got a near-empty page. Most AI crawlers don't run JavaScript, so to them these homepages are close to blank, even when robots.txt lets them in.
What should you check on your own site?
- Decide crawler by crawler. Blocking training (GPTBot, ClaudeBot) and blocking AI search (OAI-SearchBot, PerplexityBot) are different choices with different costs.
- Read your robots.txt by group. A group naming one crawler replaces your * rules for it, which is how sites block or open things by accident.
- Check your CDN's bot settings too. A firewall can block a crawler that robots.txt allows.
- Make sure the homepage text is in the HTML, not only added by JavaScript.
- Add Organization schema with sameAs links, and an llms.txt if you want to hand assistants a summary.
The free AI Crawler Checker does the first and fourth checks in a few seconds. Whether assistants then actually name and cite you is what an AI visibility audit measures.