Skip to content

AI crawlers (GPTBot, ClaudeBot, PerplexityBot)

Bots run by AI companies that download website content for model training and for answers with web search.

AI crawlers are the bots AI companies use to download web content. They are key to visibility in AI answers — if you block them, AI cannot see your content and cannot cite you.

The most important AI crawlers:

  • GPTBot (OpenAI) — training GPT models
  • OAI-SearchBot (OpenAI) — ChatGPT Search answers
  • ClaudeBot (Anthropic) — training Claude models
  • PerplexityBot (Perplexity) — the index for Perplexity answers
  • Bytespider, Amazonbot, cohere-ai and others

Tokens without their own bot: Google-Extended (Google) and Applebot-Extended (Apple) are not crawlers and never appear in the User-Agent header — they are only tokens for rules in robots.txt. The website is fetched by Googlebot or Applebot respectively; the token only decides whether the company may use the content to train its AI models (for Google, also for grounding in Gemini). It does not affect Google Search or AI Overviews — those are controlled by Googlebot.

How to control them: via robots.txt — each crawler has its own token for the User-agent line. A common mistake among Slovak and Czech online stores: a blanket Disallow rule that unintentionally blocks AI crawlers too, or a hosting/CDN firewall that rejects them at network level (in which case robots.txt makes no difference at all).

Recommendation for online stores: allow AI crawlers — for e-commerce, the business value of visibility in AI answers far outweighs the risks. Our free validator checks whether your crawlers are allowed correctly.

Does AI mention your store?The free audit shows you in about a minute.Start an audit