Cloudflare Just Changed the Rules of the Web: AI Crawlers Are Now Blocked by Default

As of Tuesday, the web works differently. Cloudflare — the infrastructure company whose network handles roughly a fifth of all internet traffic — has begun enforcing new defaults that automatically block AI training and AI agent bots on advertising-supported pages, while leaving traditional search crawlers alone. For millions of websites, the change took effect with no action required. For the AI industry, it is the most consequential redrawing of the rules of web access in a generation.

The policy was announced on July 1 with a September 15 enforcement date, and the deadline has now passed. From this week, new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers who did not adjust their settings are defaulting to a simple posture: search yes, AI training and agents no — at least on the pages where publishers make their money.

Cloudflare co-founder and CEO Matthew Prince. His company is forcing AI firms to separate search crawlers from training and agent bots — or be blocked by default. (Photo: Steve Jennings/Getty Images for TechCrunch, via Wikimedia Commons, CC BY.)

What changed on September 15

Cloudflare's new system divides automated traffic into three categories — Search, Training, and Agent — and lets site owners treat each differently. Search covers crawlers that index content for discovery. Training covers bots that vacuum up content to build or fine-tune models. Agent covers systems acting on a user's behalf in real time, from chat-fetching bots to browser-based AI assistants. Owners can allow each category outright, block it entirely, or block it specifically on pages that display advertisements.

The enforcement mechanism targets a particular offender: the "mixed-use" crawler that blends search, agent use, and training into a single bot. Starting this week, those crawlers are blocked from ad-supported pages by default unless the site owner says otherwise. The message to AI companies is blunt — split your crawlers by purpose, or lose access.

The scope is what makes this seismic rather than symbolic. Cloudflare says its network handles about 20% of internet traffic, according to Reuters. When defaults flip across that much of the web at once — including every free-tier site whose owner never touched a setting — the training-data pipeline of the entire AI industry constricts overnight.

The Google problem

No company is more exposed by the new rules than Google, and Cloudflare is not being subtle about it. The company specifically called out the "world's largest search engine" as having access to roughly twice as much information as other AI companies, because — in Cloudflare's telling — Google makes it effectively impossible to stay discoverable in search without also feeding its AI systems.

Google has pushed back on that characterization. The search giant points to Google-Extended, a control that lets publishers opt out of having content used for training and AI products such as Gemini Apps and the Vertex API, without affecting inclusion in Google Search. But there is a catch Cloudflare keeps hammering: Google's flagship Googlebot crawls for Search including AI features like AI Overviews and AI Mode. Google-Extended offers no separate opt-out for those. And blocking Googlebot at the network level would also nuke a site's conventional search presence — commercial suicide for most publishers.

Earlier this month, Cloudflare tightened the screws further. As Search Engine Watch reported on September 9, the company is removing an exemption that had kept mixed-use crawlers such as Googlebot, Microsoft's Bingbot, and Applebot outside its AI-blocking rules. Under the new policy, each crawler is assessed against all of its classified uses, and the strictest restriction wins. A bot classified for both search and training can now be blocked even on sites that allow search crawling. Site owners who want the old behavior must actively opt out.

This is not a new crusade for Cloudflare's CEO. Matthew Prince has been arguing for months that Google's bundled crawler is an unfair advantage — telling TechCrunch last year that Google effectively tells publishers "we can take [your content] and use the same crawler we use for search in order to power our AI systems. And if you want to opt out of one, you have to opt out of both." He has even pressed regulators, urging the UK to force Google to unbundle its search and AI crawlers. The September 15 defaults are the technical enforcement of that argument.

Pay per crawl becomes pay per use

Blocking is only half the strategy. The other half is Cloudflare's marketplace for the content economy it wants to replace scraping with. The company already operates Pay Per Crawl, which lets websites charge AI bots for scraping their content. Now, Cloudflare says, that is evolving into "Pay Per Use" — publishers will be able to charge AI companies when their content creates value, not merely when it is fetched.

The distinction matters. A crawl is a cost; a citation in an AI answer is revenue-generating usage. Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts in, they get paid when their content surfaces in Ceramic's AI search results or when You.com accesses their premium content. Other AI companies can customize the model, the company says.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Prince said when announcing the policy. "Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."

There is a waste argument, too. Cloudflare's data suggests that over half of all crawl traffic from AI bots is spent re-fetching pages that have not changed — pure bandwidth burned for nothing. Defaults that cut that flow save publishers real infrastructure money, whatever one thinks of the AI politics.

Why publishers are cheering — and who should be nervous

For publishers, this is the leverage they have been begging for. The last three years of the AI boom have followed a grim script for anyone who makes content: chatbots and AI search summaries answer questions directly, referral traffic collapses, and the crawlers keep coming back for more training data — all without compensation. Cloudflare has just handed every site owner a negotiating position by default, which is precisely why the defaults matter more than the settings page ever did. Most website owners never change defaults.

For the AI labs, the implications cut two ways. The giants with the deepest pockets — and the cleanest crawler separation — will adapt by paying. That is, after all, the outcome Cloudflare wants: a licensed content economy rather than a scraped one. But smaller labs and open-weight projects, already under pricing pressure from the giants' sovereign-scale spending, now face a web that charges admission. The moat around training data just got deeper, and the companies digging it are the ones who can afford the shovels.

The open question is enforcement in practice. Determined scrapers have a long history of routing around blocks — rotating IPs, mimicking browsers, ignoring robots.txt. Cloudflare's edge enforcement is harder to dodge than a text file, but the cat-and-mouse game is permanent. What changed this week is the default assumption of the web itself: for the first time at internet scale, AI companies are guilty until proven licensed.

Sources: TechCrunch (July 1, 2026); Engadget; Gulf News (Sept. 15, 2026); Search Engine Watch (Sept. 9, 2026); Cloudflare Blog; Reuters (via Economic Times).