Debugging privacy-preserving protocols is hard. Oblivious HTTP has several different steps across four different parties, not to mention binary HTTP encoding and details spread across many draft RFCs. We've taken what we've learned operating protocols like Oblivious HTTP at the scale of millions of requests per second, and wrapped it up in a nice, clean CLI tool — that we are open sourcing today.
We call it our privacy-client , or pvcli . We’re releasing it under the Apache-2. 0 License, and it is open for contributions .
Here’s a single line of code that executes a full Oblivious HTTP request with a relay, gateway and origin. Don't worry if you don't know what that means, we'll cover it below. We’ll explain why we built this tool, and show just how handy it can be.
Why privacy protocols can be hard to debug Let’s take a closer look at our motivation for creating pvcli. Over time, the Privacy team’s product suite and customer base grew. We added products like Privacy Proxy and Privacy Gateway, which power Apple’s Private Relay , Microsoft’s Edge Secure Network VPN , Flo Health’s Anonymous Mode , and more.
With it came an increasing amount of special customer requirements, domain knowledge, and complexity. As a result, we saw increased friction in development and incident response. To see this in action, let’s look at how one of our products implements Oblivious HTTP , also known as OHTTP.
First, a quick primer. OHTTP provides users with a privacy guarantee: no one can know both who made a request and what they’re requesting. To achieve this, OHTTP requires two servers, a relay and a gateway, operated by two non-colluding parties.
Below is a sequence diagram of OHTTP, where our customer owns the relay and Cloudflare owns the gateway. At a high level, OHTTP can be broken down into these steps: Client gets public key from the gateway. Client encrypts the request and sends it to the relay.
Relay removes “who” the client is from the encrypted request, and sends it to the gateway. Gateway decrypts the request, and sends it to the target. Target processes the request, and sends a response to the gateway.
Gateway encrypts the response, and sends it to the relay. Relay sends the encrypted response to the client. Client decrypts, and gets the plaintext response.
It involves quite a bit of back-and-forth, as you can see: Each step is a potential point of failure that we have to consider while debugging! In particular, we saw certain kinds of problems when debugging OHTTP. Customers asked for ways to test the live system from their end, and we often wrote one-off, custom clients for our customers specific deployments.
Figuring out which step caused an issue was time-consuming. Was the root cause a bug in our system or our customer’s system? Examining raw bits was tedious and highly prone to human error.
OHTTP builds on binary HTTP, which is a binary encoded HTTP request. Anytime we needed to check the binary encoding, we were painstakingly going through raw bits. As a result, we decided to place all of our privacy protocols in one tool.
It has a clean interface that’s already familiar, displays every single step of the protocol in order, and is flexible enough to support new protocols and architectures. To see the difference this makes, let’s see an OHTTP debugging scenario — before and after pvcli. Debugging without pvcli Say we operate an OHTTP relay that sits in front of a customer's gateway.
The customer has asked us to do an end-to-end test with a request: Recall the OHTTP steps from earlier. The first step is to fetch the public key from the gateway. We use curl to fetch it from the customer gateway and get this back: That’s a big binary string in hex.
To make sense of it, we look at OHTTP RFC 9458 §3 and parse it manually: 0029 is 41 in decimal, telling us this public key entry has 41 bytes associated with it. 55 is the public key ID. 0020 identifies the asymmetric encryption method we can use.
In this case, DHKEM(X25519, HKDF-SHA256). b9bb667e2230dc01c6d6cc047f94a1083beb185c63e50ec09f7692a5a0832540 is the public key. 0004 tells us there are 4 bytes of symmetric cryptographic IDs that follow.
0001 and 0001 identify the symmetric encryption methods we can use: HKDF-SHA256 and AES-128-GCM. We repeat this process for however many public keys are in the binary string. Next, we convert our original HTTP request into binary HTTP, referencing RFC 9292 .
We manually craft the binary with the help of some bespoke scripts: We verify each field: 02 means it's an indeterminate-length request 04504f5354 is POST 056874747073 is https 117461726765742e6f687474702e696e666f is target. ohttp. info and so on Finally, we form a wrapper HTTP request that will hold our OHTTP request.
To do so, we spend some more time writing another makeshift script that encrypts the binary HTTP request in the manner OHTTP specifies, using the public key from earlier. We create a header, which is the concatenation of public key ID, asymmetric encryption method ID, and symmetric encryption method IDs. Then, we concatenate header and encrypted binary HTTP request, resulting in: We put those bytes into the body of our wrapper HTTP request, and send it to our relay.
We get back a response. What does that mean? We reach out to the customer to ask if they can share logs from their gateway.
In the meantime, we double-check the bits we've crafted. The decoded public keys look fine. The binary HTTP request...
Oh! We see: BHTTP is length prefixed. That means we specify a length (0x0a is 10 in decimal), and then 10 bytes follow.
But here, 11 bytes follow. There is an extra 20 before the 00. 20 represents a space character, so we must have accidentally added that when building the body.
We remove the extra character, resend, and it works!
Originally published at blog.cloudflare.com
