When I last wrote about our DNS propagation pipeline in 2022 , the Zone Builder received about 250 DNS record changes per second. Today it receives about 700, nearly three times as many, plus about 100 other events, such as zone setting changes. However, the number of events was never the real problem.
The problem was the kind of work some of those events needed, and how that work grows with the size of a zone. Some zones now hold more than 20 million records, and we expect them to keep growing toward hundreds of millions. Each event starts a clock that measures how long it takes us to turn that change into data our authoritative DNS servers can use.
We call that work a build. A build reads the current database state, converts it into the format used at the edge, and writes any changes to Quicksilver . In 2022, per-record builds were 150 times faster than full builds on average.
On a test zone with one million records, per-record builds were 4,250 times faster. We ended that post with the obvious next step: do the same thing for zone settings. Why doesn't the Zone Builder just rebuild the setting that changed?
It should, and that is where we ended up. However, proving when it was safe took most of the work. A record event identifies what changed.
A zone event can come from many Cloudflare products, may not say why the zone changed, and can alter data shared by several zones with the same name. Choosing too much work wastes capacity. Choosing too little can leave stale or incorrect DNS data at the edge.
We built a targeted build planner that chooses the smallest build we can prove is safe, and a new scheduler that combines work for the same zone. Zone setting changes no longer rebuild zones with tens of millions of records. Together, they cut peak memory by 85%, and the extra headroom let us safely shorten waits the pipeline has always needed for resilience, halving the time from a DNS change to its data being written to Quicksilver.
How DNS changes reach the edge A DNS zone is the collection of records and settings for a domain. Cloudflare uses a multi-stage pipeline to turn a change to that zone into data available across our global network . A customer changes a DNS record through the DNS Records API or dashboard, or another Cloudflare service changes a zone setting.
The change is stored in a database. The database change or another system produces a Kafka event. The DNS Kafka Aggregator combines events from several Kafka topics and sends them to the Zone Builder.
The Zone Builder reads the required zone data and converts it into keys understood by our authoritative DNS servers. The Zone Builder compares those keys with Quicksilver , our distributed key-value store. It writes only what changed, and Quicksilver propagates those changes across Cloudflare's network.
This is a simplified view. The Zone Builder does not copy database rows directly to the edge. It combines records with zone settings, applies DNS serving rules, and organizes the result so our authoritative DNS servers can read it quickly.
Why full builds stopped scaling The original Zone Builder had one useful property: it did not need to know why a zone changed. It rebuilt the desired state from the current database state. Read the current zone and its settings.
Read every DNS record in the zone. Convert those records into Quicksilver keys. Compare the new keys with those already stored.
Remove stale keys, write changed keys, and update zone metadata. The final write contains only changed keys. However, a full build still reads, encodes, and compares every record, and holds the whole zone in memory while it does so.
That is cheap for a small zone. For our largest zones, with tens of millions of records, the zone does not fit in memory at all. I explain how we handle that below.
Zone settings made targeting harder. Universal SSL is one example. Enabling or disabling it updates metadata and can change Certification Authority Authorization (CAA) records at the zone apex, the bare domain, such as example.
com . CAA records tell certificate authorities whether they may issue a certificate for the domain. A full build only needed to know that the zone changed.
A targeted build must answer three questions: Which DNS names can change? Can shared metadata or the list of published zones change? What previous state must be removed from Quicksilver?
Answering those questions safely meant solving five problems, which the rest of this post covers in order: Events had to say what changed. Each change had to map to the smallest build that is still correct. Builds had to account for data shared by every zone with the same name.
The database could change between the event and the build. Combining work could not lose a Kafka message. Making events say what changed We changed the event producer to include changed zone fields and settings, with previous and new values when available.
We then traced each setting through the code that generates DNS output and classified what it could affect. Not every event was that specific. Some database functions only update a timestamp to say, in effect, "something about this zone changed."
Sometimes that is harmless bookkeeping. Sometimes it means data changed elsewhere and must be read again. Those cases look identical in the event.
Until we can identify the caller and its purpose, a timestamp-only update still causes a full build. The same rule applies to missing details, unknown events, or any change that may affect records anywhere in the zone. The planner never turns uncertainty into less work.
Choosing the smallest safe build Quicksilver output has two parts that matter here. Record keys contain the DNS answers for one zone and name. A smaller shared metadata key tells our authoritative DNS servers how to use that zone, including where to find special records such as nameservers and wildcards.
Some events change only record keys; others must update both.
Originally published at blog.cloudflare.com
