AI Search
Blocking AI Training vs AI Search Crawlers: GPTBot, OAI-SearchBot, Google-Extended and More
You can block AI training without disappearing from AI search, if you block the right user agents.
AI companies now publish separate crawlers for different jobs: some collect content for model training, others index pages for AI search results, and others fetch pages when a user asks. Blocking one doesn’t block the others.
This guide lists what each company documents, so you can choose deliberately.
Short answer: To block AI training but stay visible in AI search, disallow the training crawlers (OpenAI’s GPTBot, Anthropic’s ClaudeBot, and Google-Extended for Gemini training and grounding) and allow the search crawlers (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot) and Googlebot. Each company documents that these controls are independent. User-triggered fetchers like ChatGPT-User and Perplexity-User may not follow robots.txt in the same way.
Key Takeaways
- Training and search crawlers are separate; blocking one doesn’t block the other.
- Blocking OAI-SearchBot can remove you from ChatGPT search answers.
- Google-Extended doesn’t affect Google Search inclusion or ranking.
- User-triggered fetchers may not apply robots.txt rules.
- OpenAI says robots.txt changes take about 24 hours to take effect.
Crawlers by purpose
| Company | Training | Search / indexing | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Google-Extended (token) | Googlebot (Google Search, incl. AI features) | n/a | |
| Perplexity | Not used for training, per Perplexity | PerplexityBot | Perplexity-User |
What each company says
- OpenAI: OAI-SearchBot surfaces sites in ChatGPT search; opted-out sites won’t appear in search answers. GPTBot is for training; disallowing it doesn’t affect ChatGPT search. ChatGPT-User visits on a user’s behalf and robots.txt rules may not apply.
- Anthropic: blocking ClaudeBot signals future content should be excluded from training; disabling Claude-SearchBot or Claude-User may reduce visibility in Claude’s search and user-directed retrieval.
- Google: Google-Extended controls Gemini training and grounding; it doesn’t affect Search inclusion and isn’t a ranking signal. AI Overviews and AI Mode use normal Search crawling.
- Perplexity: PerplexityBot surfaces and links sites in its search and isn’t used for foundation-model training; Perplexity-User generally ignores robots.txt.
Example: block training, allow AI search
User-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: Google-ExtendedDisallow: /
Search crawlers not named in a group follow your general rules, so they remain allowed unless you block them elsewhere. Test changes carefully; a typo in robots.txt can block Googlebot too.
Add rules for each subdomain you want covered, and check the file with our robots.txt mistakes guide.
Choosing a policy
| Goal | Block | Allow |
|---|---|---|
| Maximum visibility | Nothing | All crawlers |
| Visible in AI search, not used for training | GPTBot, ClaudeBot, Google-Extended | Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Out of AI products entirely | All AI crawlers | Googlebot (note Google’s AI features use Search crawling) |
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot is for training; ChatGPT search uses OAI-SearchBot.
Does blocking Google-Extended affect my Google rankings?
No. Google says it doesn’t affect Search inclusion and isn’t a ranking signal.
Can I block AI tools from reading my page when a user asks?
User-triggered fetchers like ChatGPT-User and Perplexity-User may not follow robots.txt, according to their documentation.
Want your content ready for AI search?
We review how your pages, entities and structured data hold up in AI-powered search, and fix the SEO foundations those features rely on.
Sources
- OpenAI: Overview of OpenAI crawlers
- Claude Help Center: Anthropic's crawlers and how to block them
- Google Search Central: Google common crawlers (Google-Extended)
- Perplexity: Perplexity crawlers
Last reviewed October 14, 2026.
Table of Contents
Preparing for AI Search?
We strengthen the SEO foundations AI features draw on: indexable, clear, trustworthy pages.
Related Guides
Free SEO Audit
A manual on-page and technical SEO review of your website, with prioritized fixes.