Is Cloudflare blocking AI crawlers on your site? How to check (and what the defaults really are)

No — Cloudflare is not silently blocking AI crawlers on every site. Domains that existed before July 1, 2025 were never touched, and newer domains were asked at onboarding, with block as the default answer. If GPTBot or ClaudeBot are getting 403s from your site, it’s almost always one of four switches — and all four take about ten minutes to check.

Did Cloudflare really start blocking AI crawlers for everyone?

The “Cloudflare now blocks AI for everyone” phrasing is folklore; the reality is narrower. On July 1, 2025 Cloudflare called itself the first Internet infrastructure provider to block AI crawlers by default — but the mechanism was an onboarding question: “every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access,” with block preselected (Cloudflare press release). Existing domains were not changed, and an owner who picked Allow was never blocked at all.

Two more myths worth killing. The block is enforced at the network level — user-agent analysis plus machine-learning fingerprinting — not via robots.txt; the managed robots.txt is a separate, opt-in, honor-system feature announced the same day. And the appetite wasn’t new: by July 2025, more than 1 million customers had already enabled the one-click block from 2024. Cloudflare handles traffic for 20% of the web — that’s why its defaults matter.

What is the timeline of Cloudflare’s AI-crawler changes?

Six dates cover the whole arc — the 2024 one-click toggle, the 2025 new-domain default, and the 2026 preset-and-defaults overhaul.

Which four Cloudflare layers can block an AI crawler?

Four independent switches can each stop a crawler, and an Allow in one does not neutralize the others. Since July 1, 2026 the old toggle is replaced by three per-category presets — Search, Agent, Training — each settable to Block on all pages, Block on pages with ads, or Allow, on every plan including free (Cloudflare changelog).

The precedence is documented: “AI Crawl Control’s AI crawler blocking uses WAF custom rules, which take place before Cloudflare bot solutions,” with pay per crawl after them (Cloudflare docs). A crawler must clear every layer — an Allow in AI Crawl Control doesn’t exempt it from Bot Fight Mode.

Why does robots.txt say Allow while AI bots still get 403s?

In Cloudflare’s model, robots.txt is monitored, not enforced. The AI Crawl Control dashboard tracks “the number of times the AI crawler has violated your robots.txt file” as its own column, separate from the Action you set (Cloudflare docs) — because, as Cloudflare itself puts it, “robots.txt is an honor system. Nothing forces bots to follow it”. That cuts both ways: a crawler your robots.txt welcomes can be 403’d by a zone rule before your file matters.

This drift shows up in the wild. Community reports describe a Cloudflare Pages project whose *.pages.dev subdomain served AI crawlers fine while the same project’s custom domain returned “invisible” 403s — reportedly even with the owner’s bot settings on Allow, and nothing in Security Events (Cloudflare Community). Medium The block sits at the custom domain’s zone layer — a bare pages.dev project has no customer zone to configure. (Fittingly, those threads are challenge-walled by Cloudflare’s own bot protection, so we cite them as reports, not evidence.)

How do I check whether Cloudflare is blocking AI crawlers on my site?

Five checks, roughly ten minutes, no paid plan required.

1. AI Crawl Control → Crawlers tab (dashboard → account → zone). Look for a Block in the per-crawler Action column. Free plans detect crawlers by user-agent string, with 24 hours of metrics.
2. Security → Settings, filtered to “Bot traffic.” One screen surfaces the Bot Fight Mode toggle, the AI bot policy presets, and the managed robots.txt setting.
3. Security → Events. Requests challenged by Bot Fight Mode carry “Bot Fight Mode” in the Service field; filter by user agent to see how GPTBot or ClaudeBot were handled.
4. Fetch yourdomain.com/robots.txt. If managed robots.txt is on, a Cloudflare-written section sits above your own file, disallowing eight named AI crawlers including GPTBot, ClaudeBot and CCBot. See our AI crawlers reference for each bot’s operator, purpose, and verification method.
5. Run curl -I -A "GPTBot" https://yourdomain.com/. A 403 is a strong signal; a 200 only rules out user-agent-based blocks. To confirm a real crawler in your logs, OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic publishes an IP list too.

Our free AI Readiness Checker runs the outside-in checks on any URL; if ChatGPT specifically is your question, that diagnostic has its own post.

Should Bot Fight Mode be on if you want AI visibility?

Bot Fight Mode is off unless you or someone on your team turned it on — and once on, treat it as incompatible with welcoming AI crawlers. It “identifies traffic matching patterns of known bots” and answers with computationally expensive JavaScript challenges an automated crawler can’t complete. Cloudflare doesn’t publish which bots it challenges — but the free plan has no mechanism to exempt the ones you want: verified-bot allowlisting is a paid Super Bot Fight Mode feature (Cloudflare docs), and WAF rules can’t help because BFM runs in “a separate evaluation pipeline where Skip, Bypass, and Allow actions have no effect”. Cloudflare’s own docs warn about false positives against “legitimate human or automated traffic.”

We leave it off on hzsignal.com deliberately — but that’s a trade-off, not a free lunch. Cloudflare de-listed PerplexityBot in August 2025, alleging stealth crawlers evading no-crawl directives across tens of thousands of domains (Perplexity disputed it). Bot protection exists for real reasons; it and AI visibility simply pull in opposite directions. Make it a decision, not an accident.

What did the check show on our own site?

We tested hzsignal.com on July 2, 2026 — robots.txt contents, the Bot Fight Mode setting, and live curls — and found no block. Our hand-written robots.txt explicitly allows nine AI user agents (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended) with no managed prepend; Bot Fight Mode is off; curl tests as GPTBot, ClaudeBot and PerplexityBot each returned HTTP 200 — and a user-agent-matching block at any enforcement layer would have 403’d those curls. Honest caveat: a spoofed-UA curl from a residential IP only proves the absence of user-agent-based blocks — it can’t fully emulate how verified bots are handled.

A bigger caveat from our own data: being allowed is necessary, not sufficient. That same welcoming robots.txt was live while Google indexed our new pages in a median of 3 days and Bing crawled zero of 16 pages in ~11 days — no Cloudflare block anywhere in that story (one site’s experience, not a universal law). If a bot ignores you, an unblocked path is only the first thing to rule out.

What changes on September 15, 2026?

New domains onboarding from September 15, 2026 get new defaults: Training and Agent bots blocked on pages that display ads, Search allowed (Cloudflare changelog). The same day, mixed-purpose crawlers combining Search and Training become subject to Training blocks — including under the legacy “Block AI bots” toggle, which is deprecated. One genuine uncertainty: TechCrunch reported the defaults also reach all existing free customers, while Cloudflare’s own changelog says only new domains — scope beyond new domains is unconfirmed. Medium All customers can opt out before September 15.

Two related shifts: since July 1, 2026, “Verified” no longer implies allowed — per Cloudflare’s blog, “the Verified label makes a bot allowable with its relevant category,” so your per-category policy decides. And pay per crawl, still a closed beta, is evolving toward pay-per-use; Cloudflare data reported by TechCrunch says more than 50% of AI-crawler traffic re-fetches unchanged pages.

Frequently asked questions

Sources

• Cloudflare press release, Jul 1, 2025: Cloudflare just changed how AI crawlers scrape the Internet-at-large
• Cloudflare blog: Control content use for AI training · Your site, your rules · Perplexity stealth crawlers
• Cloudflare changelog: New options to manage AI traffic
• Cloudflare docs: Bot Fight Mode · AI Crawl Control · Managed robots.txt · AI Crawl Control with Cloudflare bots
• OpenAI: Bots and crawlers · Anthropic: Claude Help Center on crawlers
• Squarespace: Request that AI models exclude your site · Webflow: Set robots.txt rules
• TechCrunch, Jul 1, 2026: Cloudflare’s new policy pushes AI companies to pay for publishers’ content
• Our own tests: Experiment 01 (index-lag), run live Jul 2, 2026 — one site’s experience, not a universal law.