GLM-5.3 cross-request data exposure hid behind HTTP 200

155 GLM-5.3 responses returned other customers' data while reporting HTTP 200. The cross-request data exposure hit @cf/zai-org/glm-5.3 on Workers AI; service came back with fixes on September 9, 2026. If you run disaggregated SGLang operators, the immediate moves are checking cache capacity in startup logs and upgrading to v0.5.22 or later, per Cloudflare's incident report.

An ordinary long-context request could trigger it. Cloudflare found no evidence of deliberate exploitation. For teams serving multiple customers through shared inference infrastructure, the uncomfortable detail is that coherent output and successful status codes concealed broken isolation. The independent Python reconstruction makes the stale-memory mechanism easier to inspect. Below, a separate, scaled-down model illustrates it without requiring inference hardware.

Compare the running KV cache pool to the advertised context

The first check is the actual key-value (KV) cache capacity, not the configured context window. SGLang calculates its pool at startup using GPU memory left after loading model weights. Cloudflare says the selected size appears in startup logs and is exposed as a metric.

Read that capacity on every prefill and decode node, and compare it with the advertised context length. I'd use values from the running engine rather than assume a deployment configuration proves the pool is large enough. Also inspect the deployed SGLang dependency: Cloudflare recommends v0.5.22 or later.

This Python check uses the incident's published values; replace them with your running deployment's values when reviewing capacity.

context_tokens = 1048576
logged_prefill_tokens = 681280

print("Pool covers context:", logged_prefill_tokens >= context_tokens)

The result is false for the affected configuration. A valid prompt could fit the advertised context window while exceeding the memory available to prefill.

The pool covered only 64.97% of the advertised window. That left a 367,296-token band in which a prompt exceeded the 681,280-token pool but remained within the 1,048,576-token context limit. Those requests reached the unsafe rejection path. The distinction matters: these were not necessarily requests violating the model's advertised limit. The serving configuration accepted a larger context than prefill could actually hold.

Capacity is also sensitive to more than the context setting. Cloudflare identifies quantization, tensor parallelism and CUDA graph sizes as changes that can affect the resulting pool. I would probably treat the startup-log comparison as a deployment gate, not a one-time sizing exercise: the value to approve is the one the changed engine actually allocated.

Why rejected GLM-5.3 requests reached decode

The cross-request data exposure required both a capacity mismatch and an unsafe rejection path. Cloudflare knew the prefill pool was smaller than the context window before launch, but expected oversized requests to fail safely. Its tests checked response codes and performance, not output coherence.

In disaggregated serving, prefill processes the prompt and transfers its KV cache to decode, which generates the response. Here, rejection did not stop that handoff:

1. The scheduler rejected a prompt exceeding the pool and substituted a one-token placeholder.
2. The admission queues checked finished(). Rejection was recorded in to_finish, not finished_reason, so the request still appeared live.
3. Decode reserved pages for the original prompt. Prefill processed the placeholder, transferred one page and signalled completion.
4. Decode generated against its reserved pages, including pages this request had never overwritten.

The placeholder itself was part of an error-handling mechanism. Req.set_finish_with_abort recorded the client error and substituted the token so the request could drain through normal batch processing without allocating real memory. Cloudflare says this worked on a single-node deployment: the placeholder ran, the batch completed and the client received an error. Disaggregation broke that expectation.

Cloudflare's prefill and decode pools communicate through Mooncake using remote direct memory access (RDMA). The receiving node therefore needed an accurate account of what the sender had written. A completed transfer of the placeholder was not a completed transfer of the original prompt, but decode proceeded as though those were equivalent. Both admission queues missed the pending rejection.

Those pages could still contain earlier customers' prompts and partial generations. Cloudflare says pages were not zeroed between requests; safe reuse depended on correct ownership and lifetime tracking. This incident was not an intermittent scheduling race:

“Every request over the threshold behaved this way.”

— Cloudflare's incident report

The response metadata did not expose the mismatch. Clients received HTTP 200 rather than HTTP 400, and prompt_tokens still matched the original long prompt. Output was coherent and arrived faster than processing the full prompt should have taken. Coherence was therefore misleading: decode had usable context, but that context could belong to earlier customers.

Recreate cross-request data exposure with Python

The useful part of the reconstruction is the mismatch between memory reserved and memory written. The following model starts with pages containing one tenant's tokens, overwrites only the rejected request's placeholder, then reads the entire reservation.

page_size = 16
pages = [
    [f"tenant-a:{page * page_size + slot}" for slot in range(page_size)]
    for page in range(8)
]

# The rejected request writes its placeholder, not its original prompt.
pages[0][:1] = ["tenant-b:placeholder"]

attended_tokens = []
for page in pages:
    attended_tokens += page

stale_tokens = [
    token for token in attended_tokens
    if token.startswith("tenant-a:")
]

print("Reserved tokens:", len(attended_tokens))
print("Stale tokens:", len(stale_tokens))
print("First page:", pages[0])

The model reads 128 token slots, of which 127 still belong to the previous tenant. Even the first page retains old slots because the placeholder replaces only one list element. These are simulation counts, not measurements of the fraction of customer data in Cloudflare's affected responses.

A transferred page can still contain stale slots, so counting transferred pages alone is insufficient. The reconstruction labels slots with readable strings for inspection; the production failure involved KV state from which the model generated a response, not a direct dump of these Python-style tokens.

Timeline: fixed in September, disclosed in October

Cloudflare's dated timeline places restoration on September 9 and coordinated disclosure on October 26, 2026. The model was therefore already remediated when the reports were published. The table preserves the incident report's timeline; all times are UTC.

Cloudflare increased the prefill pool to about 2.04 million tokens, enough for the full 1,048,576-token context. It also patched rejection handling before restoring service. Cloudflare confirmed that no other Workers AI model was affected: its other disaggregated deployments had pools larger than their context lengths.

The report came from Gadi Evron of Knostic with a reproducible script. During investigation, Cloudflare also found SGLang issue #30233, filed by another user on July 6, describing this failure mode. Fix #36988 had merged on September 6 for a related path, but had not yet shipped in the release Cloudflare was running.

Cloudflare's proposed patch, #38704, addressed both admission and the receiving side: retire pending rejections before bootstrap, allocation or transfer, and verify that transferred token counts match decode's reservation. Cloudflare validated the configuration and engine changes against the researcher's reproduction and its own canary-marker tests before making the model public again.

What to patch and how to verify

Cloudflare recommends v0.5.22 or later, rather than stopping at the earliest incident fix. Its primary report distinguishes the original cross-request data exposure patches from subsequent race, lifetime and observability work.

1. Upgrade SGLang, or backport the required patches. The incident fixes include #36988 and #38935, which prevent rejected requests from proceeding into the handoff.
2. If a measured pool is too small, lower the context length to fit. Repeat the capacity check after changes to weights, quantization, parallelism or memory fraction. Cloudflare explicitly warns against relying on a proxy to enforce this limit.
3. Use deferred KV release on both roles and bound Mooncake transfer batches. Configure the load balancer to avoid unnecessary mid-transfer aborts and allow processing through the first token.
4. Validate isolation as well as error handling. Cloudflare used the researcher's reproduction and canary-marker tests. I'd make a controlled marker test part of deployment acceptance, checking that a later request never returns another test tenant's marker.

For that marker test, I would probably use independent test accounts, put a unique marker in session A, then submit the long-context test from session B and inspect B's output for A's marker. I would check the returned error as well as the output: a successful response with no marker is not, by itself, evidence that the rejection path is safe.

The release settings need checking together. Cloudflare requires deferred release on both prefill and decode, with transfer batch bounds inside the release timeout. The reason is mechanical: RDMA writes already sent cannot be recalled. A timeout must not let a page return to the pool while an outstanding transfer can still write into it.

Cloudflare also changed its load balancer's disconnect handling. It holds the prefill connection until prefill finishes and the decode connection until the first token, then closes both. Review disconnect handling alongside engine changes because aborts require coordination across both roles.

The recommended version is v0.5.22 or later. The essential patches are #36988 and #38935, which prevent rejected requests from proceeding into the handoff. The table below lists the full patch inventory with release and purpose.

Inspect the deferred-release metrics as well as startup capacity and verification logs after rollout. The metrics distinguish holds ending on drain confirmation from holds ending on timeout, revealing whether page-release coordination completes through confirmation or waits out the timeout.

What the fixes don't cover

A larger pool closes this incident's trigger, but Cloudflare warns that configuration changes can alter pool capacity and other scheduler rejection paths may exist. That is why configuration correction and engine rejection fixes are complementary, not alternatives.

Nor does sufficient capacity solve every page-lifetime problem. Cloudflare describes decode tasks sharing pages across token generation, host-memory offload, transfer reception and error handling. If one task releases a page while another still uses it, a new request can receive memory that remains active elsewhere. The incident fixes stop rejected handoffs; the lifetime and race fixes address different ways ownership can break.

Detection also has architecture limits. Cloudflare recommends --disaggregation-enable-kv-checksum and --kv-canary where supported, with logs checked to confirm verification actually happens. Broader support, including GLM 5.3, GLM 5.3 Flash and DeepSeek V4, remains work in progress.

Per-transfer checksums validate the handoff, not every later cache read. Cloudflare is working with SGLang on broader page-level ownership validation. Its earlier prototype was disabled after reliability problems; zeroing pages on release is a draft proposal, not a deployed guarantee in this report.

The deployment checklist is specifically for operators controlling disaggregated SGLang infrastructure. Hosted Workers AI customers do not manage those nodes, and Cloudflare confirmed other hosted models were unaffected. I would not treat this report as evidence that every SGLang deployment leaked data: the documented exposure required the capacity mismatch and the unsafe disaggregated rejection path together.

I'd ship the capacity gate and rejection fix together, then add supported detection and marker tests. For teams using the hosted Workers AI model, this was a vendor-remediated incident; for disaggregated SGLang operators, preventing cross-request data exposure requires checking the running deployment rather than accepting a plausible answer as proof of isolation.