On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5. 3 Workers AI model ( @cf/zai-org/glm-5. 3 ) starting on August 27, 2026.
We have remediated the issue, and we restored the model on September 9. We have confirmed that no other model hosted on Workers AI was affected. Between August 27 and September 9, a small number of customers using the GLM 5.
3 Workers AI model received responses that contained data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens. After excluding our own testing and the researcher's, the requests we identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug.
We found no evidence of deliberate exploitation. Today we are publishing this report alongside a post from SGLang, the open-source inference engine that we use, as a coordinated disclosure. This post describes the bug that caused this issue, our response timeline, and the subsequent hardening we have conducted on our production SGLang deployments.
We also detail our continued efforts with the upstream maintainers of SGLang to address this and similar bugs. What happened GLM is a text generation model from Z. ai that is built for agentic coding workflows; it is one of the many models available through Workers AI.
Gadi Evron submitted a report on September 8 through our HackerOne program stating that very long prompts sent to @cf/zai-org/glm-5. 3 on Workers AI sometimes returned text that belonged to other users' sessions. Cloudflare reproduced the issue the next morning and took the model offline within 40 minutes of declaring an incident.
We then deployed a corrected configuration and a patched inference engine, and restored service the same day. This specific report of cross-request KV cache exposure was the result of two overlapping issues. The first was a misconfiguration of the open-source inference engine, SGLang, used by Cloudflare.
We deployed GLM 5. 3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch.
We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue. The second was a bug in how SGLang handled the resulting over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and replaced with a one-token placeholder, but instead of aborting the request and returning a client error, prefill generated the KV cache for the placeholder prompt and transferred it to the decode node.
The decode node received a "transfer complete" message and assumed that prefill overwrote the KV cache from previous requests with the full prompt for this request, when in fact it left the majority of the previous state intact. The decode node subsequently generated a response while attending over stale cache pages that still held data from earlier requests.
As a result, responses to prompts longer than 681,280 tokens could contain data from other users' GLM 5. 3 sessions. Immediate remediation After disabling the model , we took the following actions to correct this vulnerability before restoring public access to it: To fix our configuration, we made the pool hold the full context.
We adjusted our GLM 5. 3 configuration to match our GLM 5. 2 configuration, which freed enough GPU memory that the prefill KV pool now holds about 2.
04 million tokens, roughly twice the 1,048,576-token context length. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request. This alone prevents the bug from being exercised.
To fix the engine, we also made the rejection path safe. Relying on pool size alone is fragile: any change to quantization, tensor parallelism, CUDA graph sizes, or a future model version changes the pool size in a way that's easy to miss. Furthermore, there may be other situations in which the prefill scheduler can reject a request and trigger the same bug.
To prevent this, we also patched the engine so that a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. We proposed this upstream as #38704 , which also had the decode node verify that the number of tokens transferred matches the number it reserved.
The fix was already on SGLang main in #36988 , merged September 6 for a related failure path but not yet released. The maintainers also merged #38935 , which rejects these requests earlier, following our approach. We validated both changes against the researcher's reproduction script and our own canary-marker tests before restoring the model.
Background: how we serve GLM 5. 3 Workers AI runs GLM 5. 3 with SGLang in a disaggregated configuration: a pool of prefill GPUs processes the prompt and produces the model's KV cache for it, then hands that cache over remote direct memory access (RDMA), via Mooncake , to a pool of decode GPUs that generate the response.
We have written about this architecture before in our posts on high-performance LLMs and smaller, faster, safer models . The KV cache in GPU memory is a shared pool of fixed-size pages. Each request is assigned pages when it is admitted and releases them when it finishes.
Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it. This approach is common among inference engines and is necessary to achieve the best performance, but it means that the inference engine is responsible for tracking ownership of these memory pages.
Originally published at blog.cloudflare.com
