AI crawlers

Why it matters: the three jobs are governed by different tokens, so blocking one bot rarely does what you think — you can stay out of training and still be cited live, or block training and lose search visibility by accident.

AI crawlers are the bots that AI companies use to fetch your pages — and they do three different jobs: train a model, build a search index, or fetch one page live when a user asks. The three jobs are governed by separate user-agent tokens in robots.txt, which is why “I blocked the AI bot” almost never means what people think. Block the training crawler and you can still be cited live; block the wrong token and you vanish from an answer engine you wanted to be in. And allowing them is only the entry ticket: every bot below is explicitly allowed on this site, yet Google indexed 10 of our 16 test pages in a median of 3 days while Bing crawled 0 of 16 (our index-lag data). Below is the full 2026 reference, verified against each operator’s own documentation.

The reference table

Purpose is one of train (feeds model training), search-index (builds the index an answer engine cites), or live-fetch (fetches a single page in real time on a user’s request). “Respects robots.txt?” reflects each operator’s own stated behavior for that token.

How to read the three purposes

Train, search-index, and live-fetch have completely different consequences for you — and different tokens.

• Train crawlers copy content into a model’s weights. They send you no traffic and produce no citation. Blocking one (GPTBot, ClaudeBot, CCBot, Bytespider) removes your content from future training runs; it does nothing for text already trained on.
• Search-index crawlers build the index an answer engine reads from when it composes a live answer. This is the one that earns citations. If you want to appear in ChatGPT search, Perplexity, or Bing/Copilot answers, these are the bots to allow.
• Live-fetch agents (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) fetch a single URL in real time because a user pasted a link or asked a question that needs it. Because a human triggered the request, several operators say these agents may ignore robots.txt — OpenAI states ChatGPT-User “rules may not apply,” and Perplexity says Perplexity-User “generally ignores robots.txt.”

The practical trap: the token that trains on you and the token that cites you are different bots. Blocking GPTBot is not the same as blocking OAI-SearchBot, and treating them as one is the most common robots.txt mistake I see.

The “-Extended” tokens are not crawlers

Google-Extended and Applebot-Extended fetch nothing. Each is a robots.txt control token with no user-agent of its own. Google-Extended governs whether content already crawled by Googlebot “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI — and Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” Applebot-Extended works the same way for Apple: it decides whether content that Applebot already indexed is eligible for Apple’s generative-AI training, and blocking it does not affect Siri or Spotlight visibility. If you want to opt out of AI training but keep classic search, these tokens are the lever — not a Disallow on the search crawler itself.

How to verify a bot is genuine (IP ranges / reverse DNS)

A user-agent string proves nothing — it is one HTTP header, and anyone can set it. Operators warn about impostors: Common Crawl explicitly notes “crawlers falsely identifying themselves as CCBot” exist. Two reliable checks:

1. Published IP lists. OpenAI, Anthropic, Perplexity, Common Crawl, and Amazon each publish a machine-readable JSON of the IP ranges their bots crawl from (linked in the table). Match the request’s source IP against that list; if it isn’t in the range, the user-agent is lying.
2. Reverse-then-forward DNS. For Googlebot, Bingbot, Applebot, and Meta’s crawlers, do a reverse DNS lookup on the source IP, confirm the hostname ends in the operator’s domain (e.g. crawl-NNN.googlebot.com, search.msn.com, applebot.apple.com), then do a forward lookup on that hostname and confirm it resolves back to the same IP. Both directions must agree — a forged reverse record alone won’t survive the forward check.

Do this before you make blocking decisions on log data: a large share of “AI bot” traffic in raw logs is spoofed, and blocking a real search-index crawler because an impostor annoyed you is how sites accidentally leave an answer engine.

A note on Bytespider

Bytespider (ByteDance) is the one entry I can’t source to an operator doc. ByteDance publishes no official crawler page, so its purpose (training data for its LLMs) and its robots.txt behavior come from third-party observation, not a primary statement. Multiple independent reports say it continues fetching paths a site has Disallowed. Treat its “respects robots.txt: no” as reported, not operator-confirmed — and if you want it out, IP/firewall blocking is the only reliable route, since a Disallow it ignores does nothing.

HZ’s own robots.txt welcomes these bots

For the record: hzsignal.com/robots.txt explicitly Allow: /s the search-index and live-fetch bots (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended). Being citable is the whole point of this site — allowing the crawlers is the entry ticket, and getting them to actually crawl a new domain is a separate fight we track in our index-lag experiment.

Sources: OpenAI — bots documentation · Anthropic — does Anthropic crawl the web · Perplexity — crawlers · Google — common crawlers · Apple — about Applebot · Common Crawl — CCBot · Meta — web crawlers · Amazon — Amazonbot