Last month, on our second Content Independence Day, we announced a couple of features designed to give website owners more visibility and control over automated traffic: BotBase added a searchable directory of known bots to the Cloudflare dashboard, while Business Insights helped owners understand how crawlers interact with their content. We know that the ecosystem of bots is vast, making it all the more important for site owners to be able to manage bot traffic sustainably.
But this ecosystem goes both ways. While website owners need to decide which automated traffic they allow, bot operators need a clear way to identify themselves, explain what their bots do, and keep that information current. BotBase works best when both sides can participate.
When we launched BotBase, we said we would build tools to bring bot operators into this ecosystem. Until now, their experience largely ended at submission. After pressing submit, an operator had no easy way to check the submission's status, understand why it was rejected, or update an existing entry.
Today, we start to change that with the launch of BotBase for Operators, tackling what bot operators need first: transparency. A new home for bot submissions Imagine you’re a bot operator looking to submit your bot to BotBase . Where on the dashboard would you look for such a submission form?
Previously, the form lived under Manage Account → Configurations , which tied the bot clearly to your account, but didn’t acknowledge its connection to the bots ecosystem. Starting today, the bot submission experience has a home next to the rest of your bot and trust tools: Protect & Connect → Application Security → BotBase (new!) All customers can access this today directly from the Cloudflare dashboard .
Here, we’ve split BotBase for Operators by use case: Bots directory — browse, search, and filter the bots Cloudflare already tracks (the same catalogue you can explore on Cloudflare Radar ). Submission form — submit a new bot. Submission history — track everything you have submitted.
Finding BotBase solves the "where" problem. The "what happens next" problem is the one that we’ve heard is deeply important to bot operators, so we’ll cover that in the rest of this post. See where your submission stands We spoke to many bot operators, and the resounding feedback was this: submitting a bot feels like a black box .
You fill in the form, press submit, and wait, with no way to tell whether anything happened next. Now, the Submission history tab shows every bot submitted from your account, each with a clear status: Waiting for review — we have received your submission and it is in our queue. Accepted — we have reviewed it and your bot is now tracked in the directory.
Rejected — something in the submission needs to change. We tell you why, with steps you can act on, so you can fix it and resubmit. Open any submission to see its full details.
If it was rejected, you will see the reason why. If it was accepted but we adjusted how your bot is classified, you will see what we changed. Previously, operators would need to email support just to ask whether their bot got reviewed or to check on their submission's progress.
That's exactly the gap we’re closing with this new tab. Today, the submission form is no longer a black box. Every operator can now view the record of every bot they've submitted starting from today’s launch, with a status you can check anytime.
We also provide a way to filter “My bots,” from the Bots directory screen, so you can see all bots that have been submitted under the account with which you’re currently logged in. Keep your bot's information up to date A bot's identification details can change over time. You might redesign your website and end up hosting your IP list at a new endpoint.
Or you might move from an IP allowlist to signing your traffic with Web Bot Auth , and need your entry to match. Before today, the only way to reflect either change was to fill out the whole form again and submit a brand-new entry. Now, you can edit a submission you have already made.
You can also cancel a submission that is still waiting for review. We encourage every operator to keep their bot's information current. Accurate details are a key component of how a bot earns and keeps Verified status, which increasingly determines whether sites across Cloudflare's network can easily allow it based on its behavior.
Of course, it is ultimately up to the individual site owner to decide what traffic is allowed and what is not. A submission form built on an updated, pragmatic taxonomy Picture a bot. Maybe it only crawls pages to build a search index.
Maybe it also acts on a user's behalf, or pulls in data for something else entirely. How it uses what it reads matters just as much as what it does. The new intake form asks you to describe your bot the way it actually behaves.
It follows the same behavior and content use model we introduced on July 1 , so instead of squeezing your bot into a single label, you now tell us three things. First, what your bot does. Maybe it only does one thing, like indexing pages for search.
Maybe it's an agent acting on a user's behalf, or it collects data, trains models, or supports SEO tools. You can select every behavior that applies, not just the closest match. Second, how it uses what it reads.
A crawler that skims a page for a search snippet is not the same as one that stores that page to train a model. You tell us the level of content use your bot needs, using the same Content Signals model website owners already use to set their own rules. For example, a site's robots.
txt might read Content-Signal: search=yes, ai-train=no, use=reference, telling every crawler it's fine to index the page for search and keep a reference, but not to train a model on it. Your bot's content-use declaration is what gets checked against exactly that kind of preference. Third, who's actually running it.
Originally published at blog.cloudflare.com
