CCBot
CCBot is the web crawler run by the Common Crawl Foundation, a non-profit that publishes a free open repository of web-crawl data — the dataset many AI companies use as raw material for training large language models.
Why it matters: your content can reach dozens of AI models indirectly through Common Crawl even if you never allow those models' own crawlers — CCBot is the shared upstream source, and one Disallow covers it.
CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/) and is operated by the non-profit Common Crawl Foundation, not by any single AI company. Crawler access is upstream of everything we measure here: with a fully open robots.txt, Google indexed 10 of our 16 glossary pages in a median of 3 days while Bing crawled 0 of 16 (our index-lag data). CCBot’s crawl feeds a public dataset used widely as training input across the industry, so allowing it can put your content into many models at once — and blocking it (User-agent: CCBot / Disallow: /) cuts that shared upstream. Common Crawl publishes its IP ranges at index.commoncrawl.org/ccbot.json and notes impostors exist, so verify by IP or reverse DNS (*.crawl.commoncrawl.org) rather than trusting the user-agent. See the AI crawlers reference. (Source: Common Crawl — CCBot.)