GEO Cited
Book a call

Lourdes Paul Agilan · 2026-07-18

The Technical Checklist for AI Citation: robots.txt, llms.txt, and Schema for SaaS Sites

Most technical GEO advice circulating right now overstates how much of this is confirmed by the AI companies themselves. This post separates what OpenAI, Anthropic, Perplexity, and Google have actually published from what's community convention or educated guesswork, with the specific directives, file formats, and sources, so you can implement the real ones without spending time on the decorative ones.

The three things that actually matter: the only technical requirements any AI company has confirmed are (1) don't block their crawlers in robots.txt if you want to be findable, and (2) make sure your content is actually readable without JavaScript. Everything past that (llms.txt, most schema types, AI-specific meta tags) ranges from "genuinely useful for entity clarity" to "no confirmed effect at all." We'll flag which is which as we go.

robots.txt: the crawlers, named precisely

Each AI company runs more than one bot, and they serve different purposes. Blocking one doesn't block the others. Getting this wrong either shuts you out of citation entirely or opts you into training data collection you didn't mean to allow.

OpenAI publishes four, each controlled independently in robots.txt (OpenAI crawler docs):

The distinction that matters most: you can block GPTBot (opt out of training) while leaving OAI-SearchBot untouched (stay visible in ChatGPT's actual citations). A lot of "block all AI bots" robots.txt templates circulating online do both by accident.

Anthropic runs three, documented at support.claude.com (last updated Feb 20, 2026): ClaudeBot for bulk training crawls, Claude-User for on-demand fetches, and Claude-SearchBot for search/answer quality. Anthropic's bots also respect the non-standard Crawl-delay directive if you want to throttle rather than fully block.

Perplexity runs two (docs.perplexity.ai): PerplexityBot, which explicitly states it won't index text from any site that disallows it in robots.txt, and Perplexity-User, a real-time, user-triggered fetch. Perplexity's own help center acknowledges that Perplexity-User previously had a loophole where it could be prompted to fetch and summarize a blocked page anyway. They say that's been closed, though blocked sites may still surface as a bare domain, headline, and summary rather than full text.

Google-Extended is not a separate crawler. It's a robots.txt-only token that rides on Googlebot's existing fetches, and it exists purely to opt content in or out of Gemini training and grounding. Google states this directly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google crawler docs). Blocking it has zero effect on your visibility in AI Overviews or regular Search. It only controls Gemini.

Common Crawl (CCBot) is a nonprofit archive crawler, not owned by any AI lab, but its dataset is a widely used pretraining source across the industry. Common Crawl's own site doesn't state this explicitly; it's established by common knowledge of what's in various model cards.

Meta is the outlier. Meta-ExternalAgent exists and is documented on Meta's developer pages, but unlike the other three companies, Meta never announced it. A spokesperson only confirmed its purpose after journalists asked, saying simply that Meta trains on "publicly available" content (Fortune, Aug 2024). If you're auditing your robots.txt against "official" documentation, know that Meta's is the thinnest of the four.

llms.txt: useful in principle, unconfirmed in practice

The llms.txt spec (llmstxt.org) is a real, well-defined format proposed by Jeremy Howard/Answer.AI: an H1 with your site name, a blockquote summary, free-form context sections, and H2 sections that are literally markdown link lists to further resources, placed at your site root.

No AI company has confirmed reading it, and that's the part most technical GEO content glosses over. Google has said so explicitly: Search Central's own AI-features documentation states you don't need "new machine readable files, AI text files, markup, or Markdown" to appear in AI features, because Google doesn't use them. Neither OpenAI, Anthropic, nor Perplexity's crawler documentation mentions llms.txt at all.

Ahrefs tested this empirically across 137,210 domains: 28% had published an llms.txt file, and 97% of those received zero bot requests to it during the study window. Of the small number of requests that did happen, only about a fifth came from anything AI-related (Ahrefs llms.txt study).

Publish one anyway. It costs almost nothing, it's a clean place to describe what you do in plain language, and if adoption changes, you're already there. Just don't budget it as a citation strategy. It currently isn't one.

Schema.org: real value, but not the value most people claim

Google's own guidance is direct on this too: "There's also no special schema.org structured data that you need to add" for AI Overviews or AI Mode eligibility (Google Search Central). The same structured-data-matches-visible-content standard as regular Search applies.

That doesn't mean schema is worthless. It means its value is entity clarity, not a citation hack:

The most defensible framing: no AI company has published a case study or statement confirming schema markup increases citation odds. Do it for the reasons schema has always been worth doing: clarity, disambiguation, legitimate rich-result eligibility. Not because it's a documented GEO hack.

The parts that actually are confirmed

Two technical factors have real, checkable evidence behind them, independent of AI-company statements:

JavaScript rendering. AI crawlers largely don't execute client-side JS. In a documented test, ChatGPT was asked to retrieve content from a JS-rendered page and told the user directly that it "could not read the content of the page because it relied on JavaScript-based rendering" (case study covered via gsqi.com, cross-referenced against Vercel's research). If your pricing page, docs, or comparison pages render client-side only, that content is likely invisible to these systems. Server-side rendering, static generation, or a prerendering fallback specifically for bots is the practical fix.

Sitemaps and freshness. Bing has published AI-search-specific guidance here directly: accurate lastmod timestamps in ISO 8601 format reflecting genuine content changes (not sitemap-regeneration time), paired with IndexNow for faster discovery of updates (Bing Webmaster Blog, Jul 2025). This is the most concrete, dated, company-sourced technical guidance we found anywhere in this research. It's worth prioritizing over the more speculative items above it.

The actual checklist

  1. Confirm you're not accidentally blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot in robots.txt. Check each independently; don't rely on a single wildcard rule.
  2. Decide deliberately whether to allow or block the training-only bots (GPTBot, ClaudeBot, Google-Extended). This is a training-data decision, separate from your citation visibility.
  3. Verify your key pages (pricing, comparisons, docs) render meaningful content without JavaScript, or serve a static/prerendered version to bots.
  4. Publish accurate lastmod dates in your sitemap and consider IndexNow if you're on a platform that supports it.
  5. Publish an llms.txt file. It's cheap and harmless, not yet proven to matter, but positions you for if that changes.
  6. Keep your Organization schema accurate and your sameAs profiles populated, for entity clarity, not as a citation hack.
  7. Don't over-invest in FAQPage/HowTo schema specifically expecting an AI-citation return; Google has moved away from surfacing these, not toward it.

Google is the only company that's stated plainly there's no special technical trick required. Nobody has published the actual weighting of these factors for citation specifically. Treat anything more precise than what's listed above as inference, not confirmed fact.

Sources cited: OpenAI, Anthropic, Perplexity, Google Search Central and Google crawler documentation, Ahrefs, Bing Webmaster Blog, Fortune, gsqi.com/Vercel.