Cloudflare Just Changed the Rules of the Web: AI Crawlers Are Now Blocked by Default
As of Tuesday, the web works differently. Cloudflare — the infrastructure company whose network handles roughly a fifth of all internet traffic — has begun enforcing new defaults that automatically block AI training and AI agent bots on advertising-supported pages, while leaving traditional search crawlers alone. For millions of websites, the change took effect with no action required. For the AI industry, it is the most consequential redrawing of the rules of web access in a generation. The policy was announced on July 1 with a September 15 enforcement date, and the deadline has now passed. From this week, new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers who did not adjust their settings are defaulting to a simple posture: search yes, AI training and agents no — at least on the pages where publishers make their money. Cloudflare co-founder and CEO Matthew Prince. His company is forcing AI firms to separate search crawlers fro...