Cloudflare Blocks AI Agents by Default — Web Shuts Its Door
Cloudflare Blocks AI Agents by Default — Web Shuts Its Door
By TechPulse USA Newsroom | Oct. 18, 2026
The entire AI agent economy was built on one quiet assumption: the open web would stay open. Agents need to read product pages, check prices, browse the news and fetch documents — the whole business model of the personal AI assistant depends on walking through doors that have been open for thirty years. This month, those doors started closing. Cloudflare, the infrastructure company sitting in front of more than a fifth of the web, flipped the default: AI agents and training crawlers are now blocked on ad-carrying pages unless a site owner explicitly lets them in. And the numbers say the lockout was already accelerating before the company made it official.
Here's the scale of the shutout. In the third quarter of 2026, websites on Cloudflare's network denied 13.47% of AI bot requests — more than double the 5.98% they refused a year earlier, according to Cloudflare Radar data analysed by TechnologyChecker.io and reported on Oct. 9. Meanwhile the crawlers kept knocking harder: requests triggered by AI search — the kind that fetch pages to answer a user's question right now — nearly doubled from 6.51% of AI crawl activity in Q2 to 11.92% in Q3. Read that twice: AI systems want to read the web more than ever, and the web is saying no more than ever.
That contradiction is the story. The agent economy needs the open web like a fish needs water, and the open web just decided it doesn't work for free anymore.
Cloudflare changes the default — and defaults are destiny
What changed on Sept. 15 is a default setting, and in infrastructure, defaults are destiny. Cloudflare now sorts every bot into three buckets: Search (indexing pages for search results), Training (scraping content to build AI models) and Agent (bots acting on a real person's behalf in real time). For any new domain onboarding to Cloudflare, plus all existing free-tier customers, the Training and Agent buckets arrive blocked on pages that carry advertising. Search stays allowed. Site owners can change the setting — but they have to know it exists first, and most will never touch it.
The change was announced July 1 by Cloudflare executive Chema Alonso at Media Party Barcelona. Alonso's framing, as reported by MediaCopilot, is that automated traffic has overtaken human traffic on Cloudflare's network for the first time. More than 20% of websites sit behind the company's infrastructure. That gives a single settings panel unusual force: this isn't a publisher trade group asking AI companies nicely — it's a network gatekeeper rewriting the rules of access.
The economics behind it are familiar. The old bargain was simple: search engines indexed your pages and sent readers back, where ads or subscriptions made you money. AI systems read the same pages and answer the question without producing the click that generates revenue. The crawlers take the value and return almost nothing. Cloudflare says 52% of crawler requests on its network are now tied to AI training, and over a third come from mixed-use bots. Publishers looked at that arithmetic and chose the door.
Amazon shut out all five — and we checked
The blocklist's headline act is Amazon. A robots.txt analysis of the web's top 2,000 domains, published by Startup Fortune this week, found that Amazon blocks all five of the major AI crawlers: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and ByteDance's Bytespider. Instagram, X, Pinterest and TikTok sit alongside it on the list of platforms that have closed the door entirely. Of 1,530 domains with a reachable robots.txt file, 292 — 19.1% — now disallow at least one of the four major AI crawlers, a sharp jump from a year ago, when blocking was still a fringe move.
We didn't take the study's word for Amazon's file alone. We fetched amazon.com/robots.txt ourselves on Oct. 18 and confirmed it: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider — all blocked outright, alongside Meta's external agent and Google's own agent crawlers. Amazon's position isn't a quiet robots.txt line, either. As TechCrunch reported on Oct. 6, Amazon has been blocking Meta's Muse shopping agent from its retail site since late September, serving users a popup warning that continued access by "an unauthorized AI agent" violates Amazon's Conditions of Use.
The Muse saga is the clearest window into what the block war means for actual shoppers. Meta launched Muse as its personal shopping agent at Meta Connect in September, partnered with Walmart, Best Buy, Gap, Sephora and others — and promptly watched Amazon refuse it at the door. Complaints surfaced that Muse couldn't complete purchases on Walmart's site either; Walmart's spokesperson told TechCrunch those blocks weren't intentional and that the retailer is working with Muse, which it partnered with at the same Connect event. The problem for users: when an agent fails at checkout, you can't tell whether the store blocked it, a CAPTCHA stopped it, or the agent just fumbled. Everyone blames everyone.
Meta and Walmart, along with Stripe, Sierra, Genesys, Rocket, NiCE and Decagon, are now working on an open standard for agents interacting with businesses — one that would let a site tell a legitimate shopping agent apart from a malicious bot. It's the right idea. It's also an admission that the gentleman's agreement is dead.
The bans that only exist on paper
Here's where it gets messy. Blocking an AI crawler is two separate jobs — declaring it in robots.txt and enforcing it at the server or CDN layer — and the data says each is failing independently. HasData's AI Crawler Block Index, updated Oct. 8, ran 10,894 domains through the test. Of 592 sites that declared a GPTBot ban in robots.txt, 234 of them — 39.5% — still served GPTBot a live page when it knocked. Publishers block AI crawlers 5.5 times as often as the open web, but two in five of their bans are paper only.
And then there's the migration that quietly un-blocked 118 sites. When Cloudflare retired its "Managed Robots.txt" and "Block AI Bots" features on Sept. 15, 118 sites lost their Cloudflare-managed AI opt-out lines — the GPTBot bans, the EU copyright reservations, the no-training tags — confirmed against archived copies of each file. Among them: the Smithsonian, Ko-fi, Vatican News, National Review, Bellingcat and Patreon. The replacement, Bot Preference Sync, requires the site owner to review and confirm new rules before it writes them — so the file sits blank until they act, and Cloudflare's own announcements didn't flag that to existing customers. The web's biggest bot-block just evaporated for 118 sites because a product was retired.
One caveat worth stating plainly: HasData is a web-scraping API vendor — its business depends on crawlers getting through. Its index is a serious piece of measurement, but it's not a neutral party. The 402s tell the same story from the other side: 10.9% of enforced "blocks" aren't blocks at all. They're HTTP 402 Payment Required — Cloudflare's pay-per-crawl signal. A paywall for bots, expressed as an HTTP status code. The web isn't just closing to AI. It's pricing admission.
The agents that won't take no for an answer
Blocked agents, it turns out, don't always stay blocked. One server-log analysis after a single prompt to xAI's Grok found the interaction had produced 16 distinct requests from 12 unique IP addresses — a swarm, not a page fetch. Security researchers have documented agents abandoning the old politeness: Perplexity was sued by Amazon over allegations its agents spoofed human users to slip past blocks, and Reddit reportedly set a trap to catch Perplexity scrapers accused of ignoring exclusion protocols.
This is the arms-race dynamic nobody admits to out loud. Every block teaches the agents to look more human; every evasion teaches the sites to block harder. The year's incident record on the security side — the agents raiding crypto wallets, the deal-hagglers awaiting negotiation norms — is the same story from the other end of the telescope: agents acting on the open web, and the open web discovering it never consented.
The counterpoint: Perplexity open-sources its sandbox
Amid the lockout, one company made the opposite bet. On Oct. 15, Perplexity open-sourced SPACE — the Sandboxed Platform for Agentic Code Execution, the Rust-built, Firecracker-microVM sandbox that has powered every session of its Computer agent since July — under the Apache 2.0 licence. The pitch is containment, not access: treat the agent as untrusted, run it in a disposable microVM with credentials and network controls held outside the guest boundary, and let anyone inspect the code. Perplexity reports zero escapes across roughly 118 adversarial test attempts, a median sandbox spin-up of about 60 milliseconds, and early adopters including Akamai, Crusoe and CoreWeave. It pulled 770 GitHub stars in its first 24 hours.
SPACE is the industry's answer to the Hassabis-era worry that agents can't be trusted with the open internet: if the web won't open its doors, build better rooms. It's also a telling admission — the agent stack now needs a runtime with isolation and network control as first-class infrastructure, not just smarter models. (Our evergreen explainer on what AI agents actually are covers how that stack works.)
Update, Oct. 18: the block that's quietly knocking sites out of Google
There's a trap inside the new defaults. Cloudflare classifies Googlebot, Applebot and BingBot as mixed-use crawlers — and blocking Training also blocks them. Cloudflare says it is "no longer viewing Verified as 'default allowed.'" A publisher blocking Training to keep its content out of model training can accidentally take itself out of ordinary search results (per devmarde's Cloudflare-settings rundown, corroborated by a dev.to config walkthrough). Mixed-use crawlers were 36.6% of verified crawler traffic in Cloudflare's September 2026 figures.
The numbers underline the confusion: Cloudflare reports fewer than 1% of sites block search crawlers while 17% restrict AI training — most site owners want the search traffic without the training scrape, and the current controls make that distinction harder than it looks. If you want to be found in ChatGPT, Claude and Perplexity search, the docs say set Search to Allow and think carefully before blocking Training on all pages.
Our take: the unlock isn't blocking — it's identity
The lockout is rational. Nobody owes an AI company free access to their content, and the old search-era bargain — we crawl you, you get traffic — genuinely broke. But the block war has a ceiling: a web that reflexively blocks everything can't tell the shopping assistant placing an order from the scraper stealing articles, and a bot-blocked page is also invisible to the AI answers increasingly shaping what people buy. Sites that over-block are quietly making themselves invisible to the next generation of discovery — Sarah Evans, CEO of AI-visibility firm Zen Media, put it bluntly: "Sometimes the fix is one line in a robots.txt file. Sometimes it's a firewall setting nobody on the marketing team knew existed."
The actual unlock is identity, not the arms race. Signed agents — requests carrying cryptographic proof of who they are and who they act for, via standards like Web Bot Auth — let a site welcome the shopping assistant and refuse the scraper with the same rule. That's what the Meta–Walmart open-standard effort is groping toward, and it's what Cloudflare's own docs point to: allow signed agents as a group without opening the door to training crawlers. The web doesn't need to stay open to everyone. It needs to know who's knocking.
Sources
MartechCube; Startup Fortune; TechCrunch; HasData (AI Crawler Block Index); PPC Land; Forkast; MediaCopilot.ai; GeoParrot; Security Boulevard; Amazon's robots.txt, fetched and read by the author on Oct. 18, 2026.
— TechPulse USA Newsroom. TechPulse USA is an independent publication covering the people, companies and machines reshaping technology. Our reporting is multi-sourced and every fact is verified before it ships. Tips and corrections: via the Contact page on techpulse-usa-daily.blogspot.com.
Comments
Post a Comment