// Reference
AI Crawler Reference
A plain-English guide to the 14 crawlers that decide whether ChatGPT, Perplexity, Claude, Google AI Overviews, and Microsoft Copilot can cite your site. User-agents, what each one actually powers, and the distinction most teams miss: training bots are not the same as citation bots.
The most common mix-up: blocking GPTBot does not remove you from ChatGPT citations — those run through OAI-SearchBot, which is a separate user-agent you almost certainly want to allow.
// Search / Citation Bots
Allow these — they decide whether your site can show up in AI answer citations at all.
- OAI-SearchBotOpenAIOpenAI's crawler for ChatGPT search citations. Distinct from GPTBot (training). Allow it to appear in ChatGPT answers. Allow
- PerplexityBotPerplexityPerplexity's primary search crawler. Bursty traffic, listicle-favoring, requires edge caching to survive viral spikes. Allow
- Claude-SearchBotAnthropicAnthropic's live-search crawler. Powers Claude's web search citations specifically. Allow
- GooglebotGoogleGoogle's primary crawler. Powers both classical search and AI Overviews — the largest single source of AI citations. Allow
- bingbotMicrosoftMicrosoft's crawler. Powers Bing Search and feeds Copilot citations across Windows, Edge, and Microsoft 365. Allow
- FacebookBotMetaMeta's citation crawler. Powers Meta AI citations across WhatsApp, Instagram, and Facebook chat. Allow
// User-Initiated Fetch Bots
Allow these too. They fire the moment a user pastes your URL into an AI chat; blocking them creates visible failures for real visitors.
// Hybrid (Search + General Fetch)
Allow. These serve both live citations and on-demand tool fetches, so blocking either side hurts you.
// Training-Only Bots
Optional. Block only if you want to opt out of model training — it will not remove you from citations by the same vendor.
- GPTBotOpenAIOpenAI's training-data crawler. Blocking it opts out of GPT model training but does NOT block ChatGPT citations. Opt-out only
- anthropic-aiAnthropicAnthropic's training opt-out user-agent. Like Google-Extended — blocking it opts out of training without affecting citations. Opt-out only
- Google-ExtendedGoogleGoogle's Gemini training opt-out. Does NOT affect Googlebot or AI Overviews — only opts out of training. Opt-out only
- Meta-ExternalAgentMetaMeta's Llama training crawler. Aggressive and historically less robots.txt-compliant than other major vendors. Opt-out only
- CCBotCommon CrawlCommon Crawl's open-dataset crawler. Indirectly feeds training for nearly every major LLM. Optional block
Check your robots.txt
Run /check on any URL and we’ll flag the bots you’re accidentally blocking. It’s the most common reason sites quietly disappear from AI citations — and the easiest one to fix.
Run a free scan