Bytespider
Bytespider is ByteDance's web crawler, widely understood to collect training data for the company's large language models — and the AI crawler most often reported to ignore robots.txt.
Why it matters: ByteDance publishes no official crawler doc, and independent researchers report Bytespider fetching paths a site has disallowed — so if you want it out, a robots.txt rule it ignores won't do it.
Bytespider is operated by ByteDance (owner of TikTok) and typically appears in logs with a user-agent ending compatible; Bytespider; spider-feedback@bytedance.com. Crawler behavior is countable, not a matter of opinion: on our own open-robots site, Google indexed 10 of 16 new glossary pages in a median of 3 days while Bing crawled 0 of 16 (our index-lag data). Bytespider is the crawler where even that kind of accounting breaks down. Unlike the training crawlers from OpenAI, Anthropic, or Common Crawl — all of which document robots.txt compliance — ByteDance provides no primary crawler page, and Bytespider is repeatedly reported to disregard robots.txt. That’s the one practical thing to know: if you want it gone, the Disallow: / it may not honor is not enough, so block by IP range or at the firewall/WAF instead. See the full AI crawlers reference for how the compliant bots differ.