Cloudflare has set a firm deadline for the AI industry to separate web crawlers used for traditional search—such as those powering Google Search—from those employed for AI agents and model training. Starting September 15, 2026, the company’s default settings will block “mixed-use” crawlers from accessing any pages that display ads, as announced on Wednesday. This means crawlers that combine search, agent functionality, and training will be blocked from crawling these sites by default, unless site owners manually adjust their settings. The new defaults will apply to new Cloudflare customers, new sites created by existing customers, and all existing free-tier users, according to the company.

This move could significantly affect how AI model providers access web content for training and to support their agentic services. Cloudflare notes that most website owners want their content to remain discoverable through search and even via AI services, but they also seek protection against having their intellectual property given away without compensation. The company specifically highlights the “world’s largest search engine”—clearly referencing Google—as having access to “about 2x more information” than other AI companies, because the search giant makes it difficult for customers to stay discoverable without also being used for AI purposes. Google has previously pushed back against this characterization, pointing out that it offers a bot called Google Extended, which allows site owners to opt out of having their content used for training and AI products like Gemini Apps and the Vertex API. Using this opt-out does not affect a site’s inclusion in Google Search. However, Google’s flagship Googlebot still crawls for Search, including AI features such as AI Overviews and AI Mode.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince in his announcement, referring to the recent milestone where bots surpassed human traffic online for the first time—a shift not expected until next year. “Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training,” Prince added.

While Cloudflare offers numerous products to help users launch their own AI systems, the company has also released a range of tools to give publishers more control over their content in the AI era. In recent years, Cloudflare launched tools to combat AI bots, including a marketplace that lets websites charge AI bots for scraping, called Pay Per Crawl. That initiative is now evolving into “Pay Per Use,” the company said, which will allow publishers to charge AI companies when their content creates value, not just when it is fetched. This change could also help conserve publishers’ bandwidth and computing resources for AI model providers, as Cloudflare’s data indicates that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.

To put this into action, Cloudflare is initially working with two partners: Ceramic.ai and You.com. When a publisher opts in, they receive payment when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content. Other AI companies can customize this model to fit how they operate, Cloudflare says.