The short answer
Cloudflare has decoupled “show up in search” from “feed AI training,” and made blocking the default for the crawlers that refuse to tell the two apart. As of September 15, 2026, a new “Disallow AI Training” setting lets site owners keep their pages indexed by search engines while refusing to let mixed-use crawlers use that content to train models. On new domains, new sites, and free-plan accounts, training and agent retrieval are now blocked by default on ad-supported pages, and mixed-use crawlers that will not declare their purpose are blocked outright there.
The practical reading: the web is moving from “one bot, one decision” toward declared-purpose crawling, where search, agent, and training access are separately negotiable. That is good news if you publish content and bad news if your product silently depends on scraping ad-supported publishers. Teams building AI agents and RAG pipelines should assume live web coverage will keep tightening and design for it now, not after a retrieval index quietly goes stale.
What Cloudflare changed
Until now, a site owner using Cloudflare faced a blunt choice: allow a crawler or block it. That worked when a bot did one job. It broke down as the same crawler started doing several — indexing a page for traditional search, retrieving it live to answer an AI agent’s question, and ingesting it to train a model. Blocking such a “mixed-use” crawler to stop training also meant vanishing from search; allowing it to stay in search meant consenting to training. Publishers had no way to say yes to one and no to another.
The September 15 update adds that missing middle. A new “Disallow AI Training” option publishes a machine-readable preference — effectively, “index me, don’t train on me” — that participating crawlers agree to respect. Choose it, and mixed-use crawlers run by operators that honor the signal keep crawling for search, while training-only crawlers from companies that keep those functions separate are blocked without touching your rankings. Cloudflare specifically names training crawlers from Amazon, Anthropic, Meta, and OpenAI as ones it can block cleanly, because those operators run distinct search and training bots.
Cloudflare also hardened the defaults. The existing “Block” and “Block on pages with ads” controls now apply to mixed-use crawlers — including Applebot, Bingbot, and Googlebot — which they previously did not. For new Cloudflare customers, newly added sites, and every free-plan site that has not changed its dashboard, training and agent crawlers are blocked by default on ad-supported pages, while search crawlers stay allowed. Existing paying customers keep their current configuration and can opt in when they choose. Ad-supported pages are the focus for a simple reason: that is where news, reviews, pricing, and product coverage live — exactly the content AI agents want most.
The “Accountable” bargain
The mechanism that makes “stay in search, skip training” workable is a new designation Cloudflare calls “Accountable.” An operator earns it by meeting four commitments: let site owners opt out of AI training through robots.txt or a similar standard; let them opt out of AI summaries, directly or through Cloudflare, with a stated target of early 2027; provide URL-level visibility into which pages were used for training, alongside search metrics; and guarantee that opting out of training will not hurt a site’s search rankings. Apple, Google, and Microsoft have committed to those principles, so Applebot, Bingbot, and Googlebot count as Accountable and keep their search access under the new setting.
That last commitment is the crux. The reason publishers tolerated mixed-use crawling for so long was fear: refuse training and you might quietly lose search traffic from the same company. Decoupling the two — and putting a named promise behind it — removes the hostage situation. It also nudges every crawler operator toward a standard: declare what each request is for, and honor the preference attached to it. Operators that will not separate their functions are the ones most exposed to the new default blocks.
Alongside the crawl controls, Cloudflare is shifting how money moves. Its earlier “Pay Per Crawl” charged AI companies each time a bot fetched a page. The replacement, “Pay Per Use,” rolling out with partners including Ceramic.ai and You.com, pays publishers when their content is actually used in an AI answer or result rather than merely fetched. Cloudflare says more than half of AI crawler traffic re-fetches unchanged pages, burning publisher bandwidth for no value; tying payment to use rather than retrieval is meant to fix that. CEO Matthew Prince framed the urgency bluntly: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.”
What it means for US & EU software teams
If you build products that read the live web, the headline is that free, silent, blanket scraping of ad-supported content is ending. Cloudflare sits in front of a large share of the web, and its defaults set norms. A RAG system or agent that quietly pulls from news sites, marketplaces, and review pages will see coverage erode as more domains inherit the stricter settings. The durable design is declared-purpose retrieval: run separate, identifiable crawlers for search, agent, and training; read and honor robots.txt and the new preferences; and build fallbacks to licensed feeds, first-party data, and partner APIs for sources that opt out. This is exactly the kind of resilience we bake into web and platform builds rather than discovering it when an index goes stale.
If you run a content-heavy site or SaaS behind Cloudflare, the news is mostly leverage — but only if you use it. You can now stay visible in Google, Bing, and Apple search while telling training crawlers no, and you can meter AI use of your content instead of donating it. The catch is that new sites and free-plan properties now inherit stricter defaults, so a team spinning up a fresh domain should verify that the search crawlers it wants are still allowed and that any AI access it genuinely relies on has not been switched off by default. A five-minute settings review beats a surprise traffic dip.
There is a governance layer too. For EU teams, being able to state and enforce how third parties may use your content dovetails with the consent, transparency, and data-provenance expectations running through GDPR and the EU AI Act — and the EDPB’s recent scrutiny of web scraping for AI training makes documented crawler policy a defensible position, not a nice-to-have. For US teams, provable control over what leaves your properties and how it is used is increasingly a procurement and contractual question. Either way, “we know which bots touch our content and on what terms” is becoming a standard answer you should be able to give.
What to do now
- Review your Cloudflare bot and AI crawler settings. Decide, deliberately, between search, agent, and training access. Do not inherit the default by accident — and if you run new domains or free-plan sites, confirm the stricter defaults match your intent.
- Separate and identify your own crawlers. If your product fetches the web, run distinct search, agent, and training bots with clear user agents so publishers — and Cloudflare — can allow the uses you are entitled to and you are not swept up as an unaccountable mixed-use crawler.
- Audit your data supply chain. Map which sources your RAG index and agents depend on, flag the ad-supported publishers most likely to restrict access, and line up licensed feeds, partner APIs, or first-party data as fallbacks before coverage drops.
- Honor declared preferences in code. Read robots.txt and the new training/summary signals in your fetchers, cache them, and fail closed when a source opts out. Treat compliance as a feature, not an afterthought.
- Write down your crawler policy. Document how your product crawls and how your properties permit crawling. It is the artifact auditors, regulators, and enterprise buyers will ask for — and the one that turns this shift from a risk into a trust signal.
Frequently asked questions
What changed on Cloudflare on September 15, 2026?
Cloudflare updated the default behavior of its AI crawler controls. A new “Disallow AI Training” setting lets a site stay indexed for search while refusing to let mixed-use crawlers use its content to train models. The existing “Block” and “Block on pages with ads” options now also apply to mixed-use crawlers such as Applebot, Bingbot, and Googlebot, which they previously did not. The new defaults apply to new customers, new sites, and all free-plan sites that have not changed their settings; existing paid customers keep their configuration.
What is Cloudflare’s “Accountable” crawler designation?
It is a label for crawler operators that meet four conditions: they let site owners opt out of AI training via robots.txt or a similar standard; they let owners opt out of AI summaries (directly or through Cloudflare, targeted for early 2027); they provide URL-level visibility into which pages were used for training plus search metrics; and they guarantee that opting out of training will not affect search rankings. Apple, Google, and Microsoft qualify, so Applebot, Bingbot, and Googlebot keep search access even when a site selects “Disallow AI Training.”
How does this affect AI agents and RAG systems that fetch web pages?
Agent and training crawlers are the ones most likely to be limited. On new domains and free-plan sites, training and agent retrieval are blocked by default on ad-supported pages, and mixed-use crawlers that will not declare their purpose are blocked entirely there. If your product builds a RAG index or runs agents that pull live web content, expect declining coverage from ad-supported publishers, separate your search, agent, and training crawlers so each can be identified, respect robots.txt preferences, and plan for licensed or first-party data where live crawling is blocked.
What is Pay Per Use and how is it different from Pay Per Crawl?
Pay Per Crawl charged AI companies each time a crawler fetched a page. Pay Per Use, rolling out with partners including Ceramic.ai and You.com, instead pays publishers when their content is actually used in an AI answer or result, not merely fetched. The shift reflects Cloudflare’s data that more than half of AI crawler traffic re-fetches unchanged pages, wasting publisher bandwidth, and it ties payment to the moment content creates value for the AI product.
Do I need to change my Cloudflare settings right now?
If you are an existing paid customer, your settings are unchanged and nothing breaks automatically, but you should still review your bot and AI crawler controls and choose deliberately between search, agent, and training access. New domains, new sites, and free-plan sites inherit the stricter defaults, so verify your intended search crawlers are still allowed and that any AI access you rely on has not been blocked. Teams on both sides of the crawl relationship should audit which of their own crawlers are identifiable and which publishers they depend on have begun restricting access.
Sources
Cloudflare Blog — Have it both ways: stay discoverable in search while disallowing AI training (September 15, 2026)
TechCrunch — Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Help Net Security — Cloudflare changes AI crawler access rules