The answer engine optimization checklist: every item testable with a free tool
A 19-check AEO audit, every item testable free: which AI crawlers to allow, the May 2026 FAQ rich-result shutdown, and our zero-Bing-crawl IndexNow data.
An answer engine optimization audit boils down to 19 concrete checks across three questions: can AI engines fetch your page (Find), can they extract an answer from it (Read), and should they credit it (Trust)? Every item on this checklist is testable with a free tool — most of them in one run of our own checker. The placement rules aren’t taste: 44.2% of ChatGPT citations come from the first 30% of content (Search Engine Land).
This checklist maps 1:1 to the named checks our AI Readiness Checker actually scores — same check ids, same categories, same point weights, straight from the tool’s code. New to the concept? Start with the answer engine optimization glossary entry — this page is the audit.
What does an AEO audit actually check?
An AEO audit verifies machine readability — that answer engines can fetch, parse, and attribute your page — not that you’ll be cited. The 19 items break down as one hard gate (the page must load), 17 scored checks, and one reported-for-transparency item (llms.txt, scored zero because its effect is unproven).
The checker turns those into a 0–100 score. Two failures are critical and cap any score at 25: a noindex directive, or a robots.txt that blocks every live-search AI crawler. Whether passing all of this actually moves citation rates is a separate question — our structure-vs-citations experiment is testing exactly that (readout ~mid-August 2026). Until then, treat this list as necessary plumbing, not sufficient magic.
Can AI engines fetch your page? (the Find checks)
If an engine can’t fetch the page, nothing else on this list matters — which is why both score-capping criticals live in this pillar.
| Check | Points | What to do | Free test |
|---|---|---|---|
Page returns HTTP 200 (reachable) | gate | The URL must return a normal 200 — no errors, no login wall | curl -I your URL, or run the checker |
Not set to noindex (noindex) | 8 · critical | Remove any noindex meta tag or X-Robots-Tag header | View source, search “noindex” |
robots.txt allows AI crawlers (ai_bots) | 12 · critical | Allow the live-search bots (next section) | Open yoursite.com/robots.txt in a browser |
Canonical tag valid (canonical) | 5 | Point rel=“canonical” at the page itself (or the version you want indexed) | View source, check the canonical href |
Served over HTTPS (https) | 2 | Serve every page over HTTPS | Look at the URL bar |
Sitemap discoverable (sitemap) | 3 | Declare the sitemap in robots.txt and serve /sitemap.xml | Visit both URLs |
The noindex check is critical because AI eligibility flows through the ordinary index: Google’s AI-features documentation says a page “must be indexed and eligible to be shown in Google Search with a snippet” to appear in AI Overviews or AI Mode (Google Search Central). One noindex tag removes you from all of it at once.
Which AI crawlers should you allow — and for which surface?
Allow the live-search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — plus normal Googlebot. The training crawlers (GPTBot, ClaudeBot) are a separate decision that doesn’t affect whether you can be cited today.
Each vendor documents its bots and lets you target them separately in robots.txt: OpenAI, Anthropic, and Perplexity. What each bot feeds — model training, live search, or user-requested fetches — decides what blocking it actually costs you:
| Crawler | Operator | Feeds | Blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Future training use — not citations |
| OAI-SearchBot | OpenAI | ChatGPT search | Visibility in ChatGPT search |
| ChatGPT-User | OpenAI | User-requested fetches | Live reads when a ChatGPT user asks |
| OAI-AdsBot | OpenAI | Ad landing-page checks | Nothing citation-related |
| ClaudeBot | Anthropic | Model training | Future training use |
| Claude-SearchBot | Anthropic | Claude search quality | Visibility in Claude search |
| Claude-User | Anthropic | User-requested fetches | Live reads when a Claude user asks |
| PerplexityBot | Perplexity | Perplexity search | Visibility in Perplexity |
| Perplexity-User | Perplexity | User-requested fetches | Little — it “generally ignores robots.txt rules” |
| Googlebot | Search + AI Overviews + AI Mode | Everything Google | |
| Google-Extended | Gemini training + grounding | Gemini training/grounding — not Search, not AI Overviews |
Blocking is common and mostly deliberate: Originality.ai’s tracker found 35.7% of the world’s top 1,000 websites blocking GPTBot as of August 2024. In our ai_bots check, a training-only or partial block is a warning (6 of 12 points); only blocking every live-search crawler fails as critical — because then answer engines genuinely cannot fetch the page to cite it.
Does blocking Google-Extended remove you from AI Overviews?
No — and this is one of the most repeated myths in AEO checklists. Google-Extended is a robots.txt control over whether your content is used to train future generations of Gemini models and for grounding; Google states it “does not impact a site’s inclusion in Google Search” and is not used as a ranking signal (Google’s crawler docs).
AI Overviews and AI Mode are fed by ordinary Googlebot crawling: “There are no additional requirements to appear in AI Overviews or AI Mode” (AI features and your website). The controls that actually limit AI-feature appearance are the existing snippet levers — nosnippet, data-nosnippet, max-snippet — and noindex.
One transparency note: our checker counts Google-Extended among the live-search bots because of its grounding role. Blocking it won’t touch AI Overviews, but it does stop your content being fed to Gemini at prompt time.
Can engines extract an answer? (the Read checks)
The Read checks reward pages that put the answer where models demonstrably look: early, under question-shaped headings, in self-contained chunks.
| Check | Points | What to do | Free test |
|---|---|---|---|
Content is in the HTML (rendered_html) | 10 | Ship your main content in the served HTML | View source, search for a sentence of your body text |
Title tag present & sensible (title) | 5 | 10–65 characters, describes the page | View source |
Exactly one clear H1 (h1) | 5 | One H1 that states the topic | View source, count the h1 tags |
Heading structure (headings) | 6 | At least two H2s; phrase them as the questions people ask | Skim your own headings |
Sufficient content depth (depth) | 6 | ~1,000+ words for articles; short reference pages only warn | Any word counter |
A direct answer up top (answer_first) | 5 | At least 40 words of direct answer before the first H2 | Read your first paragraph cold |
Scannable & chunked (scannable) | 3 | Include at least one list or table | Look at the page |
The answer-first weighting is externally backed: Kevin Indig’s study of 18,012 verified citations found 44.2% of ChatGPT citations come from the first 30% of content (Search Engine Land). On Google’s side, a 100-page CXL study found ~55% of AI Overview citations come from the top 30% of a page (CXL) Medium — one study, modest sample.
Question-phrased H2s earn their points in the same data: cited content was 2x more likely to include a question (Indig, via SEL). Keep each section self-contained — a chunk that still makes sense lifted out alone.
And note what rendered_html honestly tests: our tool’s results say it “analyzed the served HTML only — JavaScript-rendered content may not be seen by AI crawlers (or this tool).”
Should you still ship Article and FAQPage schema in 2026?
Yes — as machine-readable confirmation of what’s visibly on the page, not for rich results, which are gone. Most checklists are still out of date on the timeline:
| Date | What changed |
|---|---|
| Aug 2023 | FAQ rich results restricted to “well-known, authoritative government and health websites”; HowTo deprecated in two steps, fully by Sept 13, 2023 |
| May 7, 2026 | FAQ rich results stop appearing in Google Search: “This feature will no longer appear in Google Search starting May 7, 2026” |
| June 2026 | FAQ search appearance, rich result report, and Rich Results Test support dropped; the documentation itself removed June 15 |
| Aug 2026 | FAQ support removed from the Search Console API |
Sources: Google Search Central blog, Aug 2023, documentation updates and Search Engine Land.
What survives: FAQPage remains a valid schema.org type, and Google’s guidance is that you can remove the markup or leave it in place.
One precision note: nothing in Google’s 2026 notice says Google will keep using FAQ structured data to understand pages. That claim circulating on SEO blogs is an inference, not a Google statement.
Does FAQ markup itself win AI citations?
The evidence is thin and mixed — the consistent thread across sources is that the visible Q&A format drives extraction, and the markup is a machine-readability bonus. A ZipTie.dev analysis concluded “FAQ schema does not directly influence ChatGPT or Perplexity citation decisions”, and SE Ranking data cited there found pages with FAQ schema averaged slightly fewer ChatGPT citations — 3.6 vs 4.2 (ZipTie.dev) Medium .
Our schema check awards 12 points for any valid, relevant JSON-LD type — Article, BlogPosting, FAQPage, Person and friends (see schema markup) — and we still emit FAQPage JSON-LD on this very page.
Does freshness still matter for AI citations?
Show a real, honestly labeled date — and ignore the freshness multipliers circulating in AEO articles, because they contradict each other. One widely shared claim says content under 30 days old earns about 3x more citations; a competing analysis says AI-cited content averages nearly three years old. Neither traces to a primary source we could verify, so we cite neither.
What is solid is Google’s date guidance: show a prominently labeled, user-visible date and specify datePublished and/or dateModified in your Article JSON-LD. And don’t fake it: “The dates must describe the publication or update date of the page, not the stories or events described therein.” Our freshness check awards 6 points for a detectable date on article-type pages (N/A on non-articles).
Will a sitemap and IndexNow actually get you crawled?
Submission is not crawling, and crawling is not indexing — it’s per engine, and we have first-hand negative data. Google’s own sitemap doc is upfront: “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed” (Sitemaps overview). IndexNow is the push protocol for Bing, Yandex, Naver, Seznam.cz, Yep and Amazon — Google tested it in late 2021 but never adopted it — and its official FAQ warns: “Submitting a URL does not guarantee immediate indexing.”
What did IndexNow actually do for us?
Nothing detectable — zero Bing crawls in 11 days (live data). The setup: 16 new pages, sitemap declared in robots.txt and submitted in Search Console, plus IndexNow pings for 8 of the URLs. Google — which doesn’t use IndexNow — indexed 10 of 16 by day 11 off the sitemap, median 3 days.
Bing crawled none of the 16 — and Bing’s registered-sitemap list stayed empty the whole time, so the sitemap auto-discovery Bing documents simply didn’t happen for us. One site’s experience, not a universal law — the full debug is in the investigation post.
Same site, same day: already citable via Google’s AI surfaces, still invisible to ChatGPT search (which leans on Bing plus OpenAI’s own crawler).
Checklist action: declare the sitemap in robots.txt, then register it explicitly in both Search Console and Bing Webmaster Tools (both free) instead of assuming auto-discovery.
Can engines tell who wrote it? (the Trust checks)
The Trust checks are attribution plumbing: a machine-readable author, date, description, and preview so engines can credit the right entity.
| Check | Points | What to do | Free test |
|---|---|---|---|
JSON-LD structured data (schema) | 12 | Article/BlogPosting, plus FAQPage where real Q&As exist | validator.schema.org |
Named author / Person (author) | 6 | Visible byline plus Person JSON-LD with sameAs | View source, search “Person” |
Date present (freshness) | 6 | See the freshness section above | View source, search “dateModified” |
Meta description (meta_desc) | 3 | 50–160 characters, answer-first | View source |
Open Graph tags (open_graph) | 3 | og:title, og:description, og:image | View source |
llms.txt present (llms_txt) | 0 · reported only | Optional — effect unproven | Visit yoursite.com/llms.txt |
For the author check, Google’s Article structured-data guidance recommends Person markup with an author.url that uniquely identifies the author — or sameAs: “Google can understand both sameAs and url when disambiguating authors.” That’s the setup on this site — Person JSON-LD with sameAs pointing at a personal site and LinkedIn.
Entities matter beyond the byline, too: in Indig’s citation study, heavily cited text averaged 20.6% proper nouns against a typical 5–8% (SEL).
As for llms.txt: we publish one, we label it speculative, and our checker scores it zero points because there’s no confirmed effect on citations yet.
How do you score yourself?
Paste your URL into the AI Readiness Checker — free, no signup — and you get all 19 checks above with pass/warn/fail and the exact points. 80 and above is ready, 55–79 needs work, below 55 poor; noindex or a full live-search block caps you at 25 regardless.
For calibration: our own 10 structured experiment pages currently score 84–96 (median ~90) — “ready”, not perfect, and we’d rather publish the real spread than round it up. What no score can tell you is whether you’ll actually be cited. That’s the open question our live experiments exist to answer, and if the data surprises us, that’s the finding — we report it, we don’t move the line.
Frequently asked questions
Does blocking Google-Extended remove my site from AI Overviews?
Which AI crawlers should I allow in robots.txt?
Did Google kill FAQ schema?
Does IndexNow work with Google?
What counts as a good AI readiness score?
Sources
- OpenAI — Bots documentation
- Anthropic Help Center — Does Anthropic crawl data from the web?
- Perplexity — Crawlers documentation
- Google Search Central — AI features and your website
- Google Search Central — Google’s common crawlers (Google-Extended)
- Google Search Central — documentation updates (FAQ rich result deprecation)
- Google Search Central blog — Changes to HowTo and FAQ rich results (Aug 2023)
- Search Engine Land — Google to no longer support FAQ rich results
- Search Engine Land — 44% of ChatGPT citations come from the first third of content (Kevin Indig study)
- CXL — Where Google AI Overviews cite from: a 100-page study
- ZipTie.dev — FAQ schema for AI answers
- Google Search Central — Article structured data (author best practices)
- Google Search Central — Provide a publication date to Google Search
- Google Search Central — Sitemaps overview
- IndexNow — official FAQ
- Originality.ai — AI bot-blocking tracker