The AI crawlers you're blocking by accident
Most sites that never show up in AI answers didn't choose that. Someone blocked "the AI bots" to stop training, a CDN setting did it at the edge, or a staging rule never got removed.
Training and search are different crawlers
The big AI companies publish separate names for the crawler that collects training data and the one that finds pages to cite in answers. Blocking one does not block the other.
| Company | Training | Search and answers | User-requested fetches |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None (Perplexity says its crawlers don't collect training data) | PerplexityBot | Perplexity-User |
| Google-Extended (a control token, not a separate crawler) | Googlebot | ||
| Apple | Applebot-Extended (a control token) | Applebot |
OpenAI and Perplexity both note that user-requested fetches may not follow robots.txt, because a person asked for the page. Names change, so check each company's page before you edit robots.txt.
The accidents
1. Blocking everything to stop training
A robots.txt that disallows GPTBot, OAI-SearchBot and ChatGPT-User together opts you out of training and out of ChatGPT search. OpenAI says the two settings are independent: you can block GPTBot and still appear in ChatGPT search by allowing OAI-SearchBot. If training is all you object to, block only the training name.
2. Thinking Google-Extended takes you out of AI Overviews
Google-Extended controls whether your content is used for Gemini models and grounding. Google says it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are a Search feature, governed by the same Googlebot controls as the rest of Search, so blocking Google-Extended changes nothing you can see in Google.
3. A CDN that blocks for you
In July 2025, Cloudflare changed its default to block AI crawlers. That block happens at the network edge. Your robots.txt can say "welcome" while the crawler gets a 403 before it ever reads the file. Other CDNs and firewalls have bot rules that can do the same.
4. The staging rule that shipped
User-agent: * followed by Disallow: / is how most staging sites keep crawlers out. When it ships to production, it blocks every crawler that respects robots.txt, AI or not.
5. Typos
Crawlers match the user-agent line without caring about case, but they do care about spelling. GPT-Bot or Claude-Bot match nothing, so the rule you think is working isn't.
The fix
Decide per purpose, not per company: training yes or no, search yes or no. Write robots.txt to match, then check your CDN or firewall's bot settings, because robots.txt can't undo a block that happens before it's read.
The free AI Bot Access Checker (my own product) shows which AI crawlers your robots.txt allows and blocks. It reads robots.txt, so check edge blocks in your CDN dashboard too.