Cloudflare Blocks AI Crawlers by Default: What It Means for Your AI Citations

Cloudflare will block AI training and agent crawlers by default on ad-supported pages starting September 15, 2026. The change hits new Cloudflare customers, new sites of existing customers, and every free-tier site. Search crawlers stay allowed. The trap: block the wrong category and you can knock yourself out of Google, because Cloudflare treats Googlebot as a multi-purpose crawler.

Cloudflare sits in front of a large share of the web, so this is not a niche vendor setting. It decides, at the network edge, which bots reach millions of sites. If those sites go dark to AI crawlers, it changes what ChatGPT, Perplexity, and Google's AI answers can fetch and cite. This is the first time a default has flipped from "allow" to "block" at that scale, and most site owners have not read the fine print.

What exactly changes on September 15?

Cloudflare split AI bot traffic into three categories and is changing the default answer for two of them. On its July 1, 2026 announcement, Cloudflare defined the categories this way:

• Search: "any behavior that collects or indexes your content, so it can answer questions about it later."
• Agent: "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now."
• Training: "a crawler taking your content to train or fine-tune models."

Starting September 15, on pages that display ads, the Training and Agent categories are blocked by default. Search stays allowed. Cloudflare's reasoning is that an ad is a signal a site owner meant for a human to land on the page and see it, so machine harvesting of that page for training or agent tasks gets the stricter treatment. Pages with no ads are not covered by the new ad-page default.

This builds on Cloudflare's July 2025 "Content Independence Day", when it first began blocking AI crawlers by default for new domains and launched a pay-per-crawl marketplace. The 2026 change is the more surgical version: not "block all AI," but "block training and agents, keep search."

Who does this actually apply to?

This is the part that gets misread. The new default does not silently rewrite settings for every Cloudflare site on the internet. Per TechCrunch, it applies to three groups:

1. New Cloudflare customers.
2. New sites set up by existing customers.
3. All existing free-tier customers.

Existing paying customers keep whatever they have configured. Cloudflare gave them a window: if a paid customer wants to opt out of the new defaults, they can mark that in their Security settings any time before September 15. So a paid domain that does nothing keeps its current behavior. A free-tier domain that does nothing inherits the block.

If you run a small site or a side project on Cloudflare's free plan, you are in scope by default. That is the group most likely to not notice, and most likely to be surprised when AI referral traffic to an ad-supported page starts thinning out.

The three categories, and the crawlers in each

To make a real decision, you have to know which bot lands in which bucket. The category names are Cloudflare's; the crawler-to-category mapping below reflects how these user agents are documented and commonly classified as of August 2026. Training crawlers feed model weights. Search and user-fetch crawlers pull live pages, and those are the ones that produce citations.

The important line runs between training and retrieval. GPTBot and ClaudeBot build long-term training sets. OAI-SearchBot and PerplexityBot build the fresh index those assistants query at answer time. ChatGPT-User and Perplexity-User fetch a specific page the moment a user asks. Block the first group and you protect your content from training while staying quotable. Block the second and third groups and you remove yourself from the pipeline that produces cited answers. If you want the mechanics of that retrieval step, we broke it down in how ChatGPT picks its sources.

Why blocking "Training" can drop you from Google

Here is the mistake waiting to happen. Cloudflare treats Googlebot, Bingbot, and Applebot as multi-purpose crawlers, and it evaluates them under all of their behaviors at once. Google runs one crawler that feeds both classic Search and its AI features, so Cloudflare cannot cleanly separate "Googlebot for search" from "Googlebot for AI."

The consequence, in Cloudflare's own framing: if you block the Training category, multi-purpose crawlers like Googlebot get blocked too, even though the Search category is allowed. A site owner who thinks "I will just block AI training, that is the safe conservative choice" can accidentally block Googlebot and fall out of Google Search entirely. Help Net Security flagged the same tradeoff when the policy dropped.

So the training-versus-everything-else decision is not clean. Blocking training is only free if the training crawlers are single-purpose (GPTBot, ClaudeBot). For crawlers that do search and AI in one pass, blocking training is the same as blocking search. Whatever you decide, confirm that Googlebot and Bingbot stay allowed unless you genuinely want to leave those search engines, which is a decision a handful of publishers are now weighing on purpose.

Blocking a crawler is not the same as blocking a citation

Two separate things get confused constantly. Stopping a training crawler protects your content from being baked into a model's weights. It does not stop a live-fetch bot from pulling your page and quoting it in an answer. Those are different bots doing different jobs.

That distinction is the whole game for generative engine optimization. If your goal is to keep appearing in ChatGPT and Perplexity answers while opting out of training, you want to block GPTBot and ClaudeBot but keep OAI-SearchBot, PerplexityBot, and the user-fetch agents open. If you block everything with "AI" in the name, you also cut the retrieval crawlers, and your citations dry up. The training block costs you nothing in visibility. The search-and-agent block costs you the citation.

This is why "block AI crawlers" is a bad instruction to follow literally. AI crawlers are not one thing. Some read your page to answer a user right now and send a link back. Others take your page to train a model that competes with you. Treating them identically throws away the traffic while keeping none of the protection you actually wanted.

Content Signals versus real blocking: two different tools

Cloudflare gives you two levers, and they do not do the same thing. Understand which is a request and which is a wall.

Content Signals Policy is a set of three directives Cloudflare adds to your robots.txt: search, ai-input, and ai-train, each set to yes or no. search covers indexing to answer questions later, ai-input covers using your content as input to an AI answer, and ai-train covers model training. It reads like Content-Signal: search=yes, ai-train=no. This is a stated preference. As one breakdown of the policy puts it, content signals express preferences, not technical enforcement, and a crawler that chooses to ignore your robots.txt can still take your content.

Network-level bot blocking in the Cloudflare dashboard is the wall. It drops the request at the edge before it reaches your origin. That is what the September 15 default actually flips. If you want a preference honored by well-behaved bots and enforced against the rest, you pair the signal with the block.

The practical read: set your Content Signals to say what you want in plain terms, then back the ones you care about with actual blocking. Signals alone are a note taped to the door. Blocking is the lock.

The decide-what-to-allow playbook

Skip the reflexive "block everything." Work backward from what you are trying to keep. Four common positions:

• You want maximum AI visibility and do not mind training. Allow everything. Do nothing, or if you are a free-tier site, opt back in to the categories the default would block. You stay fully citable and fully crawlable.
• You want citations but not training. Allow Search and Agent (OAI-SearchBot, PerplexityBot, ChatGPT-User, Perplexity-User). Block Training (GPTBot, ClaudeBot, Google-Extended, CCBot). Keep Googlebot and Bingbot explicitly allowed so a Training block does not sweep them up. This is the position most publishers who still want AI traffic should hold.
• You want out of AI entirely but stay in classic search. Block Training and Agent, allow Search, and double-check that the multi-purpose crawlers you rely on are not caught by the Training block. Accept that you will lose most AI citations.
• You want to force licensing or leave certain engines. Block the relevant crawlers outright, including the multi-purpose ones, and pursue direct deals. This is the path Gannett, Reddit, and other large publishers are openly considering as AI Overviews cut their referral traffic, with some reporting organic Google traffic down roughly half year over year. It is a real strategy for a site with leverage and a paywall. It is a bad default for a site that needs discovery.

Whatever you pick, verify it after September 15. Check your Cloudflare bot settings, confirm which categories are blocked, and watch server logs for OAI-SearchBot and PerplexityBot hits. If those stop, your citations will follow. For measuring the downstream effect, which AI engines actually send traffic is the baseline to compare against.

What this means for your AI visibility

The default has flipped, and the flip is defensible: training and agent harvesting of an ad-supported page is a different thing from indexing it for answer engine optimization. But defaults are blunt, and this one can quietly cost you Google if you treat "block AI" as one switch.

The move for anyone who wants to keep earning citations is narrow, not broad. Protect your content from training if you want to, keep the retrieval crawlers open, and never let a training block take Googlebot down with it. The sites that win the next year of AI search will be the ones that stayed fetchable on purpose, the same way the levers that drive AI citations have always rewarded pages that are easy to reach and easy to quote. Locking the door to everything is not protection. It is just disappearance with extra steps, and it does nothing that a targeted training block would not do better.