How AI Search Actually Works: Which Engine Reads Which Index

Two things are true on a lot of sites right now. robots.txt blocks GPTBot. And ChatGPT still quotes those sites, by name, with a working link.

Nothing is broken. GPTBot collects training data. Retrieval is a different job, done by a different bot, reading a different index.

Every AI assistant is two systems wearing one interface: a model that writes, and a retrieval layer that decides what the model is allowed to read. Everyone argues about the model. The retrieval layer is the part that decides whether you exist.

Which assistant reads which index

In short: ChatGPT and Copilot are Bing-shaped. AI Overviews, AI Mode, and Gemini read Google's index. Perplexity is the only major assistant that crawls the web for its own index and admits it.

The same map as a table, because a table is what you will actually copy into a ticket:

Assistant	Index it reads	Crawler that fills the index	Bot that fetches live	Training bot
ChatGPT (search)	Bing, plus OpenAI's own crawl	bingbot, OAI-SearchBot	ChatGPT-User	GPTBot
Microsoft Copilot	Bing	bingbot	—	—
Google AI Overviews	Google Search	Googlebot	—	Google-Extended
Google AI Mode	Google Search	Googlebot	—	Google-Extended
Gemini (grounded)	Google Search	Googlebot	—	Google-Extended
Perplexity	Its own	PerplexityBot	Perplexity-User	—
Claude	Undisclosed providers	Claude-SearchBot	Claude-User	ClaudeBot
Siri / Spotlight	Apple's own	Applebot	—	Applebot-Extended
Meta AI	Meta's own	meta-externalagent	meta-externalfetcher	meta-externalagent

Two columns in that table are doing the heavy lifting, and they are not the ones most teams look at. The crawler column decides whether you are retrievable. The training column decides almost nothing about retrieval at all.

Why Bing suddenly matters again

Bing spent fifteen years as the search engine people set as a default by accident. Then two things happened.

First, ChatGPT started citing it. Seer Interactive matched 500+ ChatGPT citations against the same queries run through both engines in February 2025. Bing's top organic results matched 87% of the time. Google matched 56%, with a median rank of 17. One study, one snapshot, 100 queries — treat it as a strong signal rather than a law. But the direction is not subtle.

Second, Microsoft retired the Bing Search APIs on August 11, 2025. The whole family went: Web, Image, News, Entity, Custom Search. The replacement is Grounding with Bing Search, sold inside Azure AI Agents. Bing stopped being a search engine you query and became a grounding layer you buy.

So the index that answers your customers' questions inside Copilot is the same index that answers them inside ChatGPT, and increasingly the same one rented by every Azure-hosted agent in your competitor's stack. If you have never opened Bing Webmaster Tools, that is the highest-leverage hour in your quarter.

One honest gap: OpenAI has never published its provider list. What is documented is OAI-SearchBot, OpenAI's own crawler, described in its own docs as being "used to surface websites in search results in ChatGPT's search features." What is measurable is the Bing overlap. Plan for both.

Google runs two doors, and only one of them is a crawler

This is where most robots.txt files go wrong.

AI Overviews and AI Mode are not separate products with separate crawlers. They are surfaces of Google Search, fed by the Google Search index, filled by Googlebot. Google's own documentation is blunt about it: there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." A page needs to be indexed and eligible to show with a snippet. That is the whole entry requirement.

Google-Extended is a different animal. It is not a crawler. It has no user-agent string and it never requests a page. It is a control token that governs how already-crawled content gets used — training future Gemini models, and grounding in Gemini Apps and the Vertex AI API. Google states it plainly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews run on Search eligibility. The lever that actually limits what Google shows from your page in AI Overviews is nosnippet, data-nosnippet, or max-snippet — and those cost you regular snippets too. There is no free door.

Perplexity is the only one crawling for itself

PerplexityBot builds the index. Perplexity-User fetches a page live when a user's question demands it. Perplexity's docs are explicit that PerplexityBot "is not used to crawl content for AI foundation models."

The crawler is also the reason to care about verification. In August 2025 Cloudflare published evidence that Perplexity was reaching blocked content through undeclared crawlers presenting a generic Chrome user-agent from unrelated ASNs — millions of daily requests across tens of thousands of domains. Cloudflare pulled Perplexity from its verified bot program. In the same tests, ChatGPT-User fetched robots.txt and stopped when disallowed.

The lesson generalises past any one vendor: a user-agent string is a claim, not a credential.

The three jobs a bot can have

Here is the mental model worth keeping. Every AI crawler on your logs is doing exactly one of three jobs, and each job has a different consequence when you block it.

1. Train — reads your page into a dataset for a future model. Blocking costs you nothing today. It does not remove you from any answer.
2. Index — reads your page into a retrieval index that assistants search at query time. Blocking makes you invisible in that assistant, permanently, for every user.
3. Fetch — reads your page right now because a human asked a question that needs it. Blocking makes you the source the assistant could not open, mid-conversation, in front of the person who wanted you.

Most publishers intend to block job 1. Most publishers, reading a list of bot names and pattern-matching on the vendor, block job 2 or 3 instead — or block job 1 and believe they blocked all three.

Job	Example tokens	Blocking costs you
Train	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent	Nothing in retrieval
Index	bingbot, Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot	Every future citation
Fetch	ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher	The answer being written right now

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is training. ChatGPT's search features reach you through OAI-SearchBot and, on the evidence, through Bing. Block GPTBot and you stay fully citable.

The same shape repeats everywhere. Block Applebot-Extended and Apple states your pages can still appear in Siri and Spotlight, because Applebot is the one that crawls. Block Google-Extended and Search is untouched. Block ClaudeBot and Claude-SearchBot keeps indexing.

The inverse is the expensive mistake. Block bingbot — perhaps in a bot-mitigation rule someone wrote in 2019 — and you disappear from Bing, from Copilot, from DuckDuckGo, and from most of what ChatGPT can find.

A robots.txt that says what you probably mean

Opt out of training. Stay retrievable. Roughly:

# Retrieval — index me, cite me
User-agent: Googlebot
User-agent: bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Applebot
Allow: /

# Live fetch — a human is waiting
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: meta-externalfetcher
Allow: /

# Training — no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

Two asterisks. Google-Extended here blocks Gemini API grounding, not AI Overviews — Search keeps its access either way. And meta-externalagent does double duty as both training and indexing for Meta, so blocking it is a real trade, not a free one.

On Bing there is a second control worth knowing, and it lives in meta tags rather than robots.txt. NOARCHIVE keeps your content out of Copilot answers and grounding while leaving you in search results. NOCACHE is softer: Copilot may cite you, but only with URL, title, and snippet.

Verify the bot, not the string

Anyone can send User-agent: OAI-SearchBot. Every major operator publishes IP ranges or supports reverse-DNS verification, and that is the only signal worth enforcing on. Match on IP, then apply your policy. Rate-limit or challenge the rest.

If you are logging crawler traffic — and you should be — log the resolved identity, not the claimed one. Otherwise your "AI traffic" dashboard is a chart of what bots decided to tell you.

What this means for how you publish

Retrieval bots are not browsers. They fetch, strip your markup, and try to recover meaning from whatever survives. Every hydration boundary, every client-rendered section, every layout div between a heading and its answer is a place where your content gets thinner on the way in.

So the practical work splits cleanly. Get indexed by the crawlers that feed the assistants your audience uses — which for most teams starts with Bing, not because Bing wins, but because Bing is upstream of ChatGPT and Copilot and priced into Azure. Then make the page cheap to parse once the crawler arrives.

That second half is what we built plym around: serve markdown to agents, HTML to humans, from the same URL, in milliseconds. Same content, no reconstruction step, nothing lost between your CMS and the index.

Start with the logs, though. Pull last month's user-agents, sort them into train, index, and fetch, and check which column your robots.txt is actually acting on. Most teams find at least one rule doing the opposite of what it was written to do.

References

• Google: AI features and your website
• Google: Common crawlers and Google-Extended
• Anthropic: Does Anthropic crawl data from the web?
• Perplexity: PerplexityBot and Perplexity-User