Skip to content

robots.txt for AI crawlers: what to block and what to allow in 2026

Vladimír Donič· 20 Apr 2026 · 6 min read

TL;DR

Your robots.txt may be the reason ChatGPT, Gemini and Perplexity can't see you. 34 % of Slovak online stores have AI crawlers blocked — either deliberately (a misguided attempt to protect against scraping) or by accident (old robots templates from the pre-ChatGPT era). This article contains a complete list of AI crawlers and a ready-made robots.txt template you can use as is.


Why robots.txt is key to AI visibility

The robots.txt file at yourshop.com/robots.txt tells web crawlers which parts of the site they may index. It was originally meant for Google, Bing and other classic search engines. Since 2023, AI crawlers have entered the picture — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot (Perplexity) and others. They were joined by the Google-Extended (Google) and Applebot-Extended (Apple) tokens — these are not crawlers, just rules in robots.txt (explained below).

If your robots.txt blocks the search AI crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), the outcome is clear-cut: the AI search engine doesn't know about you and can't recommend you. You won't appear in ChatGPT Search, Perplexity or Claude's search — no matter how good your content or schema.org markup is. Training crawlers (GPTBot, ClaudeBot) are a different matter: they decide whether your content may be used to train models, not whether AI search can find you.

The complete list of AI crawlers in 2026

CrawlerAI platformBlocking it means
OAI-SearchBotChatGPT Search (OpenAI)Invisible in ChatGPT search
GPTBotOpenAI model trainingContent isn't used for training; ChatGPT Search is unaffected
Google-Extended (token, not a bot)Gemini Apps and Vertex AI (training and grounding)Google may not use your content for Gemini; Google Search and AI Overviews are unaffected
PerplexityBotPerplexity.aiInvisible in Perplexity
Claude-SearchBotClaude search (Anthropic)Invisible in Claude search
ClaudeBotClaude model training (Anthropic)Content isn't used for training; Claude search is unaffected
cohere-aiCohere (Command model)B2B AI tools
DiffbotDiffbot knowledge graphIndirect impact on AI knowledge
FacebookBotMeta AI (Llama)Meta AI recommendations
Twitterbot(legacy, now X.ai)Minimal impact
ia_archiverInternet Archive (Wayback Machine)Historical records, not AI directly

For 2026, the critical bots for visibility are the search bots OAI-SearchBot and PerplexityBot; Claude-SearchBot and FacebookBot are growing, but so far with less reach in e-commerce. There are also bots that act on a user's request (ChatGPT-User, Claude-User, Perplexity-User) — they load a page when a user asks for it in a chat, and some of them may not honour robots.txt. Sources: OpenAI Bots, Anthropic: crawlers.

Google-Extended and Applebot-Extended are not crawlers. They have no User-Agent of their own and you won't see them in access logs. The site is fetched by Googlebot or Applebot respectively; the token only determines what the company may use the already-fetched content for. Google-Extended controls use for Gemini Apps and Vertex AI (training and grounding). It has no effect on Google Search, AI Overviews or AI Mode — those are governed by Googlebot and the nosnippet, max-snippet or noindex rules. Applebot-Extended only concerns Apple model training. Sources: Google Common Crawlers, Apple Applebot.

How to check your current robots.txt

  1. Open a browser and go to https://yourshop.com/robots.txt
  2. Look for sections starting with User-agent: OAI-SearchBot, User-agent: GPTBot, User-agent: PerplexityBot (and possibly a User-agent: Google-Extended rule)
  3. If these sections are missing — AI crawlers have access (default behaviour, good)
  4. If you find Disallow: / under OAI-SearchBot, Claude-SearchBot or PerplexityBot — you are blocked from that AI search engine. Disallow: / under GPTBot or ClaudeBot only forbids model training, not search. Disallow: / under Google-Extended only switches off use of your content for Gemini, not indexing.

An example of a problematic robots.txt:

# ❌ This blocks ChatGPT Search (OpenAI):
User-agent: OAI-SearchBot
Disallow: /

# ⚠️ This forbids OpenAI model training (ChatGPT Search is unaffected):
User-agent: GPTBot
Disallow: /

# ⚠️ This forbids use of content for Gemini Apps and Vertex AI
# (Google Search and AI Overviews are unaffected):
User-agent: Google-Extended
Disallow: /

# ❌ This blocks Perplexity:
User-agent: PerplexityBot
Disallow: /

This is the template we recommend for online stores that want maximum AI visibility:

# ===========================================
# robots.txt — YourShop.com
# Updated: April 2026
# ===========================================

# --- Classic search engines ---
User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /search?

User-agent: Bingbot
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

# --- AI crawlers — ALLOWED ---
# OpenAI: OAI-SearchBot = ChatGPT Search (key for visibility),
# GPTBot = model training (optional to allow)
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

# token, not a crawler – Gemini only
User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

# Anthropic: Claude-SearchBot = Claude search (key for visibility),
# ClaudeBot = model training (optional to allow)
User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: Diffbot
Allow: /

User-agent: FacebookBot
Allow: /

# --- Unknown / harmful crawlers ---
# (add if you see excessive server load)
# User-agent: SemrushBot
# Crawl-delay: 10

# User-agent: AhrefsBot
# Crawl-delay: 10

# --- Sitemap ---
Sitemap: https://www.yourshop.com/sitemap.xml
Sitemap: https://www.yourshop.com/sitemap-products.xml
Sitemap: https://www.yourshop.com/sitemap-categories.xml

What to block — even when you allow AI crawlers

Even with open access for AI crawlers, you should block these sections (this applies to all crawlers, AI included):

Disallow: /admin/          # Admin interface
Disallow: /checkout/       # Payment process
Disallow: /cart/           # Cart — dynamic content with no value
Disallow: /account/        # Customer account
Disallow: /search?         # Search results (duplicate content)
Disallow: /compare/        # Product comparison (dynamic URLs)
Disallow: /wishlist/       # Wishlists
Disallow: /*.pdf$          # Internal PDF documents (if you don't want them indexed)

Granular control of AI crawlers

Don't want to let AI crawlers onto the whole site, only onto product pages? That's possible:

# OAI-SearchBot (ChatGPT Search) may only access products, categories and the blog:
User-agent: OAI-SearchBot
Allow: /products/
Allow: /categories/
Allow: /blog/
Disallow: /

This configuration lets ChatGPT Search index products and the blog, but not, for example, staff FAQ pages or internal documents.

Myth: blocking AI crawlers protects your content from scraping

The most common reason online stores block AI crawlers: fear that AI will "steal" their content and use it to train models.

The truth is more complicated:

  1. AI training and AI search are different things. GPTBot (OpenAI) and ClaudeBot (Anthropic) collect content for model training. ChatGPT Search uses a separate OAI-SearchBot, and Claude search uses Claude-SearchBot. You can block the training bots without losing search visibility.

  2. Training can be switched off directly in robots.txt. Disallow: / for GPTBot and ClaudeBot forbids use of your content for training, while OAI-SearchBot and Claude-SearchBot remain allowed. The result: you're in search, but not in the training data.

  3. Blocking hurts you more than scraping does. If you block OAI-SearchBot, ChatGPT Search won't recommend you to customers. That is a far bigger commercial loss than the hypothetical use of your content in training.

After changing robots.txt — what next

Once you've edited robots.txt:

  1. Wait 1–2 weeks — AI crawlers need time to revisit the site
  2. Check Google Search Console — the "Crawl stats" section shows whether Googlebot visits your site regularly (you won't see Google-Extended there, it isn't a bot)
  3. Track your AI visibility — with Pozorio AI you'll see whether your visibility in ChatGPT and Gemini improved after the change

Conclusion

robots.txt is a five-minute fix that can significantly change your AI visibility. Check it now — it's at yourshop.com/robots.txt. If you find OAI-SearchBot, Claude-SearchBot or PerplexityBot blocked there, fix it today; for Google-Extended, decide whether you want Gemini to be allowed to use your content.

Want to know whether ChatGPT and Gemini can see you at all? A free audit at pozorio.sk will tell you in 60 seconds.

Does AI mention your store?The free audit shows you in about a minute.Start an audit