Lourdes Paul Agilan · 2026-08-25

Cloudflare's September 15 AI Crawler Default Change: What It Actually Means for B2B SaaS

Cloudflare is changing a default setting on September 15, 2026. A lot of the coverage makes it sound like every site on Cloudflare needs to act before that date or risk disappearing from AI answers. That's not accurate for most B2B SaaS companies, and getting the scope wrong wastes a week of your team's attention on the wrong fix.

The short answer: the new default only fires on pages Cloudflare's classifier flags as showing ads, and only for new Cloudflare customers, new sites added by existing customers, and existing free-tier accounts. If you're on a paid Cloudflare plan with an already-configured zone, this specific change does not touch you automatically. If your site doesn't run display advertising, which covers most SaaS product sites, marketing sites, and docs, it's a non-event even if you are in one of the affected groups.

That's the correction. It's also not the interesting part. The interesting part is a separate, older setting that's been silently blocking AI crawlers on plenty of B2B sites for months, has nothing to do with this deadline, and won't be fixed by anything Cloudflare does on the 15th.

What actually changes on September 15

Starting that day, on pages Cloudflare's ad-detection logic identifies as ad-monetized (it looks for ad network tags — AdSense, Ezoic, Mediavine, and similar), two categories of crawler get blocked by default:

Training crawlers — bots that collect content to train models
Agent crawlers — bots performing real-time tasks on a user's behalf

Search crawlers stay allowed by default. Those are the bots that power live answers in tools like ChatGPT search, Perplexity, and AI Overviews — the ones that actually matter for citation.

This applies to three groups: new Cloudflare customers, new sites added by existing customers, and all existing free-tier accounts. If you run a paid, already-configured zone, nothing changes for you on September 15. You keep whatever settings you already have, and you can opt out of the new defaults in Security settings any time before then if you're in one of the affected groups and want to.

Read that against your own site honestly. If you're a martech or salestech SaaS company without display ads on your marketing pages, this specific deadline is close to irrelevant to you.

According to Cloudflare's own reasoning, an ad is a signal the page was built to be read by a person and monetized by their attention, so pages without ads don't trigger the new restriction at all, and pages with ads keep the crawlers that answer live search queries while blocking the ones scraping for model training data. It's a reasonable line to draw. It's just not the line that matters for a B2B SaaS site deciding whether to worry.

The distinction that actually decides whether you're cited

Two crawlers get confused constantly, and the confusion costs companies real citations: GPTBot and OAI-SearchBot.

GPTBot collects data for model training. Blocking it has no direct effect on whether ChatGPT can find and cite you in an answer, per OpenAI's own documentation.

OAI-SearchBot is what ChatGPT uses to fetch pages for live search answers. Block that one, and you're not showing up when someone asks ChatGPT a question your product should answer.

Companies that opted out of AI training over the past two years, a defensible, common decision, sometimes did it with a blanket rule that caught both bots under one "block AI" toggle. The result: they unknowingly opted out of being cited, while believing they'd only opted out of training. This has nothing to do with September 15. It's been quietly true since whenever that rule got added, and the September 15 news cycle is a good excuse to go check it, not the actual cause.

The real risk: your robots.txt and your edge don't necessarily agree

Here's the part that doesn't get fixed by reading Cloudflare's September 15 post and moving on.

Cloudflare, like most CDNs and WAFs, runs bot management at the network edge, separate from robots.txt. Robots.txt is a voluntary instruction that well-behaved crawlers choose to respect. A firewall rule, a bot-management product, or a "Block AI Bots" managed toggle operates before the request ever reaches your server, and it does not read your robots.txt to decide whether to honor it.

That means two settings can disagree, and nothing on your side will tell you they do. Your robots.txt can say "AI crawlers welcome," while a security rule someone enabled during an unrelated review silently returns a 403 to GPTBot, OAI-SearchBot, or PerplexityBot at the edge. Search Console, Ahrefs, Semrush, and Screaming Frog won't catch this. They render JavaScript and report on Googlebot, not on what happened to an AI crawler's request. Google keeps ranking you fine, so nothing looks wrong in the tools your team already checks daily.

The way this has actually happened to companies: a "block AI scrapers" rule gets added during a security review, nobody tells marketing, and AI citations drop to zero for months before anyone connects it back to that one settings change. When it's found, the fix is usually minutes of work. The damage was in the months nobody was looking.

This is the version of "Cloudflare is blocking AI crawlers" worth checking. It's been possible on any Cloudflare zone, paid or free, for a long time, and September 15 doesn't change it in either direction.

How to check if Cloudflare is blocking AI crawlers from your site

Three checks, in order of speed.

1. Check your robots.txt (30 seconds). Visit yoursite.com/robots.txt and look for Disallow: / under User-agent: GPTBot, User-agent: OAI-SearchBot, User-agent: ClaudeBot, User-agent: PerplexityBot, or User-agent: Google-Extended. This tells you what you're declaring, not necessarily what's happening at the edge.

2. Check what the edge actually returns. Robots.txt can say "allowed" while a WAF rule says otherwise. Confirm directly by sending a request with each crawler's user-agent string:

curl -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" -I https://yoursite.com/
curl -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" -I https://yoursite.com/
curl -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)" -I https://yoursite.com/

A 200 means the page was served. A 403, a redirect to a challenge page, or no response at all means something at the edge is stopping that crawler regardless of what robots.txt says.

3. Check your server logs for the crawlers that matter. Pull the last 30–90 days and filter for AI crawler user-agents. If OAI-SearchBot or PerplexityBot show up requesting pages and getting non-200 responses, that's your answer, and it's a specific, dated problem you can hand to whoever manages your Cloudflare zone.

If you're in one of the three groups the September 15 default applies to (new customer, new site, or existing free tier) and you do run ad-monetized pages, add a fourth check: confirm in Security settings whether the new default is set the way you want it before the 15th.

The checklist

For a B2B SaaS marketing or growth lead deciding what to actually do before September 15:

Confirm whether you're in scope. New Cloudflare customer or site, or on the free tier? The default may apply to you. Paid, already-configured zone? It doesn't, automatically.
Check for ad monetization on the pages that matter. No ads, no automatic change either way.
Run the curl checks above against GPTBot, OAI-SearchBot, and PerplexityBot, regardless of what the September 15 default does, since this checks a separate, older setting.
Pull 30–90 days of server logs and confirm AI crawlers are actually reaching your pages, not just technically permitted to.
If you find a block, fix it and note the date. Citations that recover after a fix typically show movement within one to two weeks, not months, worth tracking so you know the fix worked.
Set a recheck reminder for 90 days out. Security reviews and vendor default changes don't announce themselves to marketing. The only reliable defense is checking periodically, not trusting that nothing changed since the last time you looked.

None of this requires a tool purchase. It requires ten minutes and three curl commands. If you'd rather have someone else run this check and fix what it finds, that's the entire premise of our Citation Gap Report.

FAQ

Does the September 15 change block ChatGPT from crawling my site? Only if you're a new Cloudflare customer, a new site on an existing account, or on the free tier — and only on pages Cloudflare's classifier flags as showing ads. Existing paid, configured zones aren't automatically affected. And even then, Search crawlers — the ones that power live citations — stay allowed by default; only Training and Agent crawlers are blocked.

How do I check if Cloudflare is blocking GPTBot or PerplexityBot on my site? Check robots.txt first, then confirm at the edge with a direct request using each bot's user-agent string (see the curl commands above). Robots.txt and your actual firewall behavior can disagree, and robots.txt won't tell you if they do.

Is blocking GPTBot the same as blocking ChatGPT citations? No. GPTBot is for training and has no direct effect on search visibility per OpenAI's documentation. OAI-SearchBot is what fetches pages for ChatGPT's live search answers — that's the one that determines whether you can be cited.

What should a B2B SaaS company actually do before September 15? Confirm whether the default applies to your account (new/free-tier vs. paid/configured), check for ad monetization on the affected pages, and separately verify — using the curl checks above — that no existing security rule is already blocking AI crawlers at the edge. The second check matters more for most B2B SaaS sites than the September 15 deadline itself.

Sources cited: Cloudflare, OpenAI.