Blog
By Haobo Zhang AI SEO Jul 3, 2026 10 min read

How to check if ChatGPT can see your website: 5 tests you can run in 10 minutes

Five checks — robots.txt, a spoofed-UA fetch, Bing's url: operator, your WAF, and a live ChatGPT ask — reveal whether ChatGPT can see your site in 10 minutes.

To check whether ChatGPT can see your website: read your robots.txt for OAI-SearchBot rules, fetch a page with OpenAI’s exact user-agent strings, look the URL up in Bing with the url: operator, audit your CDN’s bot settings, and ask ChatGPT to open the page live. Five tests, all free, about ten minutes total.

“Can ChatGPT see my site” is really three separate doors, because OpenAI runs four documented crawlers: OAI-SearchBot decides whether you appear in ChatGPT search, ChatGPT-User fetches a page live when a user asks about it, and GPTBot collects training data (a fourth, OAI-AdsBot, only checks ad landing pages) — our AI crawlers reference lists every bot’s user-agent and robots.txt rule. On top of its own crawl, ChatGPT search uses third-party search providers — Bing is the one OpenAI names. The five tests below check each door in turn. They tell you whether ChatGPT can see you; whether it will cite you is a different question, and one we probe with live experiments here.

Test 1: Is your robots.txt blocking OAI-SearchBot? (2 minutes)

Open yourdomain.com/robots.txt and look for Disallow rules under OAI-SearchBot, GPTBot, ChatGPT-User, or User-agent: * — only the OAI-SearchBot token controls whether you appear in ChatGPT search. OpenAI’s crawler docs state that “each setting is independent”: blocking GPTBot is a training opt-out, nothing more, while “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

BotWhat it controlsHonors robots.txt?IP list
OAI-SearchBot/1.3Whether your site appears in ChatGPT search resultsYes — the only token that governs search inclusionopenai.com/searchbot.json
ChatGPT-User/1.0Live fetches when a user asks about your pageRules may not apply (user-initiated)openai.com/chatgpt-user.json
GPTBot/1.3Training-data collection onlyYesopenai.com/gptbot.json
OAI-AdsBot/1.0Ad landing-page validationYesopenai.com/adsbot.json

If you find and fix a bad rule, OpenAI says “it can take ~24 hours from a site’s robots.txt update for our systems to adjust” — so re-test tomorrow, not in five minutes.

Automated? Yes — our checker parses robots.txt for seven AI bots and raises a critical failure when every live-search bot (or *) is disallowed for your path.

Test 2: Does your page load when you fetch it as an OpenAI bot? (3 minutes)

Fetch your page twice — once with plain curl, once wearing OpenAI’s exact user-agent string — and compare the status codes. These are the current verbatim strings from OpenAI’s docs; note the /1.3 versions — guides still quoting GPTBot/1.0 are stale.

# Plain fetch
curl -sI https://yourdomain.com/your-page

# As OAI-SearchBot/1.3
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot" https://yourdomain.com/your-page

# As ChatGPT-User/1.0
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://yourdomain.com/your-page

# As GPTBot/1.3
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot" https://yourdomain.com/your-page

Two 200s means this door is open. But a 403 on the bot UA alone is a lead, not a verdict: Cloudflare’s fake-bot detection matches known-bot user-agents and then verifies the source by reverse DNS or IP validation — your laptop is not OpenAI, so a WAF can block you as an impersonator while waving the genuine crawler through. Confirm in your WAF or security event log how requests from OpenAI’s published IP ranges are actually treated. A 403 on both fetches points to a general WAF or geo rule instead. Fittingly, when I curled openai.com itself on 2026-07-02, I got a 403 — bot-blocking is everywhere, including on AI companies’ own sites.

Automated? Yes — our checker fetches your URL and reports any non-200, 403s included. It identifies itself as HZ-ReadinessBot/1.0 rather than spoofing anyone.

Test 3: Is your page in Bing’s index? (2 minutes)

Search Bing for url:yourdomain.com/your-page — Microsoft documents the url: keyword as the per-URL index check: “Checks whether the listed domain or web address is in the Bing index.” The popular site: operator is weaker evidence — “Returns webpages that belong to the specified site” is a site-wide sample, not a verdict on one URL. For ground truth, run URL Inspection in Bing Webmaster Tools on your verified site: it shows when a URL was discovered, crawled and indexed, plus a live check that tests crawlability “without requesting it for index.”

One documented quirk: several 2026 Microsoft Q&A threads report URLs showing “Indexed” in Webmaster Tools while site:/url: return nothing. Responders across those threads describe indexing and serving as separate processes — though the moderator in the linked thread said an Indexed page “should be displayed” — so treat this as a community-documented quirk, not doctrine. Medium If the two disagree on your page, both facts are informative.

Automated? No — per-engine index status is a manual check by nature; our checker doesn’t query Bing for you.

Why check Bing when I care about ChatGPT?

Because ChatGPT search doesn’t only rely on its own crawl — it uses third-party search providers, and Bing is the one OpenAI names (for Enterprise and Edu workspaces, Bing is “the only third-party search provider”). Consumer docs add that ChatGPT search “sometimes partners with other search providers,” and one experiment by Semrush suggests Google SERP data may be in the mix too — OpenAI hasn’t confirmed it. Medium The provider mix isn’t fully public, so the practical rule is: check each engine separately, because “indexed” is per-engine. Google indexed 10 of our 16 new pages in a median of 3 days while Bing crawled 0 of the same pages in two weeks, despite a sitemap and IndexNow pings — one site’s experience, not a universal law, but a live demonstration that Google visibility says nothing about this door.

Test 4: Is your CDN or firewall blocking AI crawlers? (2 minutes)

If your domain joined Cloudflare after July 1, 2025, check your bot settings before blaming OpenAI: since that date every new domain is asked at sign-up whether to allow AI crawlers, with block as the default posture — and Cloudflare proxies roughly 20% of the web. Existing zones kept their old settings, so this bites new sites hardest — you can be blocking bots you never chose to block. A Cloudflare WAF block returns HTTP 403 by default, and Cloudflare’s own troubleshooting doc lists WAF rules, Security Level, DDoS protection and Browser Integrity Check as possible 403 sources. Review your security events for OpenAI’s user-agents and published IPs, and check whether Bot Fight Mode or an AI-crawler block is switched on. We’ve mapped all four Cloudflare switches in a dedicated guide.

Automated? Partially — a CDN block surfaces in our checker as a failed fetch (403), but naming which rule fired is on you.

Test 5: Can ChatGPT actually open your URL? (1 minute)

Ask ChatGPT, with search enabled, to open your exact URL and quote the first paragraph verbatim — then compare the quote against your live page. This test is free: ChatGPT search rolled out to all logged-in free users in December 2024 and has required no account at all since February 5, 2025.

Three failure modes, three different meanings. If it says it can’t access the page, the fetch path is blocked — revisit tests 2 and 4. If it answers from memory without browsing, that’s evidence of nothing — insist that it open the URL. If it “quotes” text that isn’t on your page, it never really read it. For hard evidence, grep your server or CDN logs for ChatGPT-User/1.0 hits from OpenAI’s published ranges — that file was refreshed on 2026-06-24 and the ranges rotate, so re-fetch the JSON rather than hardcoding IPs. One caveat: this exercises the user-fetch door, where robots.txt “rules may not apply” — a pass here doesn’t prove you’re in ChatGPT search.

Automated? No — our checker can’t chat with ChatGPT for you.

What the free checker automates (and what it can’t)

Our free AI Readiness Checker runs tests 1 and 2 in full for any URL and covers test 4 partially (a CDN block shows up as a failed fetch); tests 3 and 5 stay manual, and I’d rather say so than oversell it. Beyond the fetch and robots.txt checks, it flags noindex directives (meta and X-Robots-Tag), missing sitemaps, canonical, HTTPS and structured-data gaps — and it raises a critical when a page is a JavaScript-only app shell, with deliberately hedged wording: “Content appears JavaScript-rendered — AI crawlers may not see it.” What AI crawlers actually execute is an open question we’re not adjudicating today; the safe move is serving your key content in the initial HTML either way.

TestAutomated by the checker?
1. robots.txt rulesYes — including the all-search-bots-blocked critical
2. Fetch and status checkYes — reports non-200s, with its own UA, no spoofing
3. Bing index statusNo — manual (url: operator + Bing Webmaster Tools)
4. CDN/WAF blockPartially — shows up as a failed fetch
5. Live ChatGPT askNo — manual by nature

Frequently asked questions

Does blocking GPTBot hide my site from ChatGPT?
No. GPTBot only governs training-data collection. ChatGPT search inclusion is controlled by OAI-SearchBot — OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. The settings are independent.
My curl test with a bot user-agent got a 403 — is ChatGPT blocked from my site?
Not necessarily. Cloudflare and similar WAFs verify real bots by IP or reverse DNS, so your spoofed user-agent can be blocked as a fake bot even when the real crawler is allowed. Check your WAF event log against OpenAI's published IP ranges.
How long after fixing robots.txt until ChatGPT search notices?
OpenAI says it can take roughly 24 hours from a robots.txt update for its search systems to adjust. Getting newly crawled and indexed can take longer.
Does ChatGPT-User obey robots.txt?
OpenAI says robots.txt rules may not apply to ChatGPT-User because its fetches are user-initiated, and it is not used to decide what appears in search. Use the OAI-SearchBot token to manage search inclusion.
Is the live ChatGPT test free to run?
Yes. ChatGPT search has been available to all logged-in free users since December 2024 and to logged-out users with no account since February 2025.

Sources