One year ago, we declared Content Independence Day . At the time, we could see what many in the industry were beginning to sense: the fundamental economics of the Internet were shifting. AI adoption was accelerating, publishers were experiencing rapid declines in referral traffic, and AI companies were crawling the web at unprecedented scale, often without clearly declaring intent, and almost always without compensation.
We changed the defaults. For all new domains on Cloudflare, AI training crawlers would be blocked by default unless domain owners chose otherwise. We didn't do this to wall off the web.
We did it because we believed a healthier ecosystem required transparency, control, scarcity, and ultimately, a market where high-quality content could be valued and exchanged fairly. A year later, that market has emerged. But the transformation of the Internet has happened even faster than we anticipated.
In this report, we share key data points that illustrate how quickly the business model of the Internet has shifted – and what this new content market means for publishers and site owners. Part I: The Internet has changed – faster than anyone expected The vertical adoption curve AI is not just another technology cycle. It is a platform shift happening at more than 2x the speed that smartphones were adopted.
In just 3. 5 years, over 30% of humanity — 2. 5 billion active users — has adopted regular use of generative AI.
The adoption curve isn't merely steep: it's going vertical. The decline of the open web Never before have we seen such a rapid change in how humans interact with information, perform work, and spend time online. The way people use the Internet is changing dramatically.
Today, for every hour spent online searching for information, only 15 minutes is spent on the open web. Traditional search behavior is collapsing as users shift to AI-driven discovery and consumption. Instead of visiting multiple sites to source and compare information, users simply type a prompt and receive a nearly instantaneous, consolidated answer.
The agentic Internet is here This year, agent traffic crossed a historic threshold for the first time: more than 50% of traffic on the Internet is now non-human. This shift has staggering implications for publishers, content owners, and the future of the open web. Crawlers have changed their purpose When looking at the crawlers Cloudflare identifies by purpose, the composition of crawler traffic tells the story clearly: 52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025.
Mixed-use crawlers (those blending search, agent use, and training) represent over 36% of activity. Pure search crawling now represents a small and declining share of overall crawler activity, despite remaining critical for publisher visibility. As AI training becomes a primary driver of crawler activity, the ability to distinguish between discovery and training becomes increasingly important.
Mixed-use crawlers blur that distinction, putting content owners in a difficult position: choose between remaining discoverable in the agentic era, and giving away their most valuable content without compensation. The old business model is gone For decades, the economic model of the open web was straightforward. Content creators exchanged access to their content for visibility in search engines, which returned referral traffic.
That traffic became the primary mechanism through which publishers, creators, and businesses generated economic value. But today, that exchange is breaking down. Content is still being crawled, indexed, and used — but increasingly without corresponding traffic being returned to the source.
As AI systems answer questions, compare products, conduct research, and complete tasks directly, information across the open web is increasingly becoming part of AI training and retrieval systems. The existential question this raises is simple: if content is consumed without audiences ever visiting the source, how do content creators sustain themselves? The implications are industry-agnostic The earliest industries to feel the impact were news organizations and media companies.
Today, similar dynamics are impacting businesses across retail, software, IT, and finance. Some of the most heavily crawled categories have seen human traffic decline as much as 40% in less than one year. Many publishers are now preparing for what they call "Google Zero" — a world where little to no traffic comes from search referrals.
The implications extend to essentially every industry. Any organization that publishes proprietary information on the Internet will need to understand how to operate in an agentic era. This dynamic matters not just to content owners, but to all of us.
The Internet is a critical part of the global economy and one of the world's most important public resources for surfacing information. Ensuring it remains healthy and sustainable is essential for all. Part II: The market has emerged What we built When we launched Content Independence Day, we committed to three things: Transparency and control for site owners, enabling them to define how their content is accessed and monetized.
Tools that create scarcity, shifting the balance of power back to content owners. A marketplace where content creators and AI companies of all sizes can discover, license, and determine the value of content more efficiently. One year later, a market for monetized content is here, and the conditions for a dynamic marketplace are forming.
Transparency and control created scarcity Historically, publishers have had limited visibility into how AI companies accessed and used their content. As referral traffic declined, that lack of visibility became an economic problem prompting publishers to seek new ways to capture value.
Originally published at blog.cloudflare.com
