Can AI engines cite content behind a login or paywall?

If your best content sits behind a login or a paywall, the uncomfortable answer is that AI engines generally can’t cite it, because well-behaved crawlers only read what’s publicly accessible. GPTBot and the other major crawlers scan public content and make no attempt to bypass paywalls, logins or restricted sections, so anything behind authentication or marked private simply stays untouched — and content that can’t be read can’t be quoted in an answer. A common misconception compounds this: people assume robots.txt is what controls access, when in fact it doesn’t authenticate anyone or block anything at the network level. It’s a request that polite crawlers honour, while real enforcement — rate-limiting, blocking, paywalls — happens at the server or CDN through WAF rules or 403 responses.

It’s worth being honest that not every crawler behaves. In Q4 2025, 13% of AI-bot requests bypassed robots.txt, up 400% from the second quarter, and Cloudflare went as far as de-listing Perplexity as a verified bot after catching it stealth-crawling with a disguised browser user-agent. But those are the exceptions that prove the rule, and you shouldn’t design a citation strategy around misbehaving bots. The real takeaway is a trade-off: citation requires exposure, so if a page’s value is locked away you are effectively choosing revenue protection over AI visibility. That can be entirely reasonable — just go in knowing that gated content won’t earn citations, and that anything you genuinely want referenced needs at least its key, quotable substance in public HTML.