We’re constantly building for the different goals of our customers. Some customers want to optimize for discovery, while others want to protect their content with the strictest security policy. Among these differing policies, there are multiple ways to mitigate bot traffic.
Some mechanisms simply state your preference, assuming best intent from crawlers, and other approaches actually lock down content by outright blocking with a Bot Management solution. We recognize that it's cumbersome to maintain multiple layers of protection on your website. For example, there are cases in which your robots.
txt states that a crawler is Disallowed from accessing your website, while your enforcement rules actually don’t block that crawler. When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules. A couple of years ago, Cloudflare announced an easier way to disallow AI training on your website by tackling two of these layers: a managed value of robots.
txt that told a fixed list of major Training crawlers not to train on your content, along with edge-enforced blocks to Training crawlers. On July 1, 2026, we launched easier options to manage different kinds of AI traffic use cases. You can say what you want to do about Search, Agent, and Training traffic on your website.
We're announcing Bot Preference Sync , available to all customers from the Free tier to Enterprise. Bot Preference Sync reflects what you've set in your AI bot configuration by updating corresponding preferences to your robots. txt, and it can be turned on or off at any time.
No more static file for one use case: we'll help you tailor your robots. txt to reflect what you’ve already configured for different AI bot categories . New questions facing the Internet For years, the most pressing question in this space was: "Is my content being used to train AI models without my permission?"
It's an important question, and it isn't going away. Alongside this, the questions we increasingly hear are about discoverability and engagement. How do I show up when someone asks an AI assistant something my site can answer?
How much of my traffic is coming from AI crawlers versus real people? What content is actually driving referrals, and what is it worth? The answers differ by business model.
Discoverability and engagement are key, top-of-mind issues for any businesses trying to thrive on the modern web, but the funnels for these are different: an e-commerce store may want everything crawled and trained on, so its products surface when a shopper asks a chatbot for "the best sofa for a small apartment." A publisher that monetizes pages with ads may want the opposite: stay in the search index that sends readers to the page, but keep its articles out of model training and, crucially, be able to verify that its content really wasn't used without permission.
There's no single right answer, which is exactly the point. Your controls should reflect your strategy, which is why we've been building tools to give you visibility and choice at every layer. Bot Preference Sync ties these together, so the preference you set is the preference you publish .
The call for Transparency On July 1, 2026, we made the case that mixed-use crawlers, or “bots that blend search, agent use, and training behind a single user agent,” put site owners at a disadvantage precisely because they make it hard to separate what you want from what you don't. That's still true, and our position on Transparency for site owners hasn't changed.
But there’s more than one way to approach Transparency. We want to reward the operators who are clear about their identity and how they are using the data they crawl. For purposes of bot Verification , the owners of bots that perform both Search and Training will need to provide additional information in order to not be blocked when “Disallow Training” is set.
Those requirements are: The bot must respect, via any mechanism, a “no training” preference in robots. txt They give site owners a way to opt out of AI summaries. They provide URL-level visibility into which pages were made available for training, as well metrics on search results, so you can see how your content was used for search and for training.
They can show publicly that Disallowing Training does not hurt your traditional search results. Bots of leading AI models and service providers that meet these criteria are tracked publicly in the AI bot transparency section in Cloudflare Radar, which includes examples in which best practices are honored, as well as when they are not. Crawlers that don't provide Transparency will not get the benefit of the doubt — they're still blocked when you disallow training.
In other words, this is a way of making Transparency the price of admission. Introducing Bot Preference Sync Bot Preference Sync is a new feature that keeps your robots. txt reflecting the AI bot preferences you've already set for Search, Agent, and Training on the Cloudflare zone-level dashboard.
If a site owner already has a robots. txt file, the contents added by Bot Preference Sync will be prepended to the existing material, so any existing Disallow directives are maintained. Instead of a site owner maintaining a separate static file, Cloudflare generates or updates your robots.
txt based on your configuration, so what you say to the world and what you enforce at the edge are kept in sync. For Search and Agent, the three options we announced on July 1 remain: Allow, Block on pages that serve ads, or Block everywhere. For Training , we’re refining the option to stop your content being used for training models with the Disallow option: Disallow : a "no training" preference is written to your robots.
Originally published at blog.cloudflare.com
