Traditional database architecture assumes you have one or two big database clusters per application, usually accompanied by a connection pooling solution. AI-assisted development and vibe coding platforms turn this model inside out and are driving applications toward many small databases: one per user, application, preview environment, or tenant. That model fits D1 particularly well.
D1 is Cloudflare's managed, serverless database based on SQLite. It allows you to create millions of databases and only pay for what you use. The question becomes: how do you scale the control plane for millions of databases without making every query slower by sending it through centralized bottlenecks?
We made a lot of architectural changes to scale out D1 to support platforms with millions of created databases, hundreds of thousands of which are active at any point. This post is a behind-the-scenes deep dive into how the D1 control plane works. Many of the architectural patterns we will describe can be used by anyone building on top of Cloudflare’s Developer Platform.
The post also gives a glimpse into how we build our own control planes. Growth in numbers There was already a steady growth in D1 usage when we started planning these changes, but we knew we would need to support a much bigger scale once large platforms started building with D1. Here is how things have played out so far.
D1 has passed 5. 5 million databases and has seen a 15x growth in the number of live non-deleted databases over the past one year , and a 65x growth over the past 2 years (as of October 8, 2026). We have also seen 18x growth in the number of daily active databases over the past year (as of October 8, 2026).
A database is considered active if it executed at least one query during that day. Established and successful platforms are already building with D1, using the same tools available to all Cloudflare users. "ChatGPT Sites lets anyone turn ideas into shareable experiences.
D1 gives our users a simple, scalable way to add features like custom metrics, leaderboards, and comments to their dashboards, games, and tools." - Jon Abrams, ChatGPT Sites @ OpenAI While we were making the changes described in this blog post, we periodically load-tested against a target well above our expected growth. This guided where to focus our optimization efforts.
Guiding engineering principles Applications built on the Cloudflare Developer Platform range from full-stack applications to service APIs, from internal single-user portals to massive multi-tenant vibe coding platforms, and everything in between. While building out the D1 control plane to support millions of databases, the following tenets and principles were guiding our architecture decisions and are useful for anyone building their control and data planes on our platform.
Design all processes with failure and idempotency in mind. Failures will happen, so the system needs to self-correct where possible and continue making progress. Avoid centralized single-region components in any hot path for data plane operations (in our case all query requests).
Use Workers KV , a globally distributed KV store, for critical routing metadata so that cache misses only rarely need to query central databases. Use Hyperdrive to access central Postgres databases for efficient connection pooling and optimized routing and caching, using read replicas wherever possible. Periodically precompute aggregations in the database to avoid scan queries during user request processing.
Use eventual consistency and queuing between components that don’t need strong consistency to increase reliability and availability. Load-test the application with realistic workloads to discover bottlenecks and inefficiencies, and focus on optimizing those first. There are more engineering principles we follow internally to build our systems, but the above are generic enough and apply to anyone building on Cloudflare.
The following sections will dive into the main system components and processes of the D1 control plane regarding the lifecycle of a D1 database, and showcase how to put these principles into practice with a real-world system. High-level architecture Let’s start off by introducing the main components of D1. There are a few more arrows and boxes missing from the diagram for secondary processes supporting D1, but these are the ones providing D1’s public feature set.
We will explore the rest in dedicated sections in the rest of the blog post. First of all, each D1 database is backed by one SQLite Durable Object , which makes it very easy to create millions of them. The “HTTP API” shape represents all our API surfaces including the Cloudflare dashboard, Wrangler, and Cloudflare CLI.
All API requests go through Cloudflare’s edge gateway and are served by the D1 Edge API Worker. The diagram shows the HTTP endpoints that allow users to execute SQL queries against their databases as the “Query endpoints”. The remaining endpoints include the usual database operations like creating, deleting, and listing databases, along with import/export and time travel features.
These are shown as the “CRUD endpoints”. The CRUD endpoints use Hyperdrive to query a Postgres cluster in our central data centers (referred to as “Core”). There are two configurations.
One configuration has caching enabled and targets the Postgres read replicas. The other configuration has caching disabled and targets the Postgres primary. The Edge API Worker might invoke the D1 Eyeball Worker (depending on the type of request).
The Eyeball Worker is our proxy for all requests that need to be handled by the underlying D1 Durable Objects (DO). This allows us to put all query handling inside the Eyeball Worker, including observability, automatic retries, error handling, and other query-specific behaviors. Workers KV acts as our caching layer at the edge for read-heavy metadata and can be read by all the D1 Workers.
Originally published at blog.cloudflare.com
