Rendered at 22:04:06 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jacquesm 20 hours ago [-]
This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible.
This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
kflansburg 20 hours ago [-]
At no point do we claim that it is a bug with the model. We specifically outline how it was a mis-configuration on Cloudflare's part and a bug in SGLang. We name the model in the title because our customers need to know which of our endpoints was affected. In fact, we say "the GLM-5.3 Workers AI model", not "GLM-5.3".
jacquesm 20 hours ago [-]
Not buying that. You explicitly mention the model name in the title the 'GLM 5.3 workers AI model' suggests it is something specific to GLM 5.3 which it is not. You mention Sglang and are all surprised that it didn't throw an error. But the issue was yours all along.
margalabargala 20 hours ago [-]
Yeah, their title is absolutely implying a GLM bug to anyone who doesn't read it super closely.
There's a huge amount of unnecessary extraneous details in the headline that distract from "this is a Cloudflare bug".
literallyroy 19 hours ago [-]
Not sure if the title was edited but as is it is very clear to me this is a bug on their side and I have no idea how a technically literate person (someone using the model via cloudflare?) who would misinterpret given the reference to the cache.
infecto 9 hours ago [-]
The title is poorly worded. I can easily see how it can be conflated in passing. Naming the model is silly in the title. It should be mentioned in the article but as read the title suggests it’s something related to this model.
XYen0n 19 hours ago [-]
I have read this post, and it appears that this is indeed an issue specific to the GLM 5.3 model on their platform. I am unsure whether this is because only GLM 5.3 uses SGLang or because only its configuration is problematic.
paimapi 8 hours ago [-]
Did we read different articles? None of the issues they brought up (architecture splitting, memory transport, Cloudflare's misconfigured context window provisioning, the SGLang state leak, lack of client error provisioning) are model related. These all exist at the layer of the harness (ie Cloudflare's configuration).
A better title would be "Cross-request KV cache exposure in Cloudflare provisioned GLM-5.3 Workers AI model" because it narrows culpability. Just posting it on the Cloudflare blog isn't enough for that in the age of AI and SEO data-scrapes and it'd be surprising to me if the authors didn't know that.
fg137 8 hours ago [-]
They could have removed GLM 5.3 from the title, and the title and the body would still make perfect sense.
neuronexmachina 7 hours ago [-]
I think it's in the title because it's also doing double-duty of informing customers that only usage of the GLM-5.3 model was potentially impacted. I agree that it's so what confusing, though.
robrenaud 14 hours ago [-]
I was surprised in reading the article that it had nothing to do with the contents of the model. It could have been any model capable of 1M context that triggered this bug/misconfig on Cloudflare/SGLangs side. The title is extremely misleading.
nijave 7 hours ago [-]
Changing "model" to "endpoint" might help since it's the endpoint, not the model
fg137 8 hours ago [-]
What's the point of even mentioning GLM-5.3? That's a detail that only belongs in the body.
Did you ever given any thought about how confusing the title may be?
Did AI write this title and you just use it without ever wondering if it makes sense at all?
asdfsa32 19 hours ago [-]
"Cross-request KV cache exposure in X model".
Which part of that is not claiming the exposure is in X model? Cloudflare is stopping to be cool, this is the start. I guess, cf is now big enough for enshitification to start.
anramon 6 hours ago [-]
> We name the model in the title because our customers need to know which of our endpoints was affected
Then you would used "endpoint" instead of "model". In fact, there's actually zero reasons to put any model or endpoint in the title if the bug is yours alone.
infamouscow 7 hours ago [-]
You should change the title.
Most people will read this headline and assume Cloudflare is intentionally trying to squash open-weight models. Given how superfluous the model is to the content, it's easy to believe that narrative.
If that wasn't your intent, you really ought to change the title ASAP. Time is ticking before tech journalists and social media decide to run with that narrative and make your public relations department sad.
nullc 16 hours ago [-]
Then remove GLM-5.3 from the title.
fg137 8 hours ago [-]
Not what I am seeing. not sure if there is A/B testing here.
jarym 12 hours ago [-]
Their AI won’t let them
nijave 7 hours ago [-]
It does have to do with GLM-5.3 but you're correct it doesn't have anything to do with the model.
It effects the _endpoint serving GLM-5.3_
I'm not sure how else you'd identify the GLM-5.3 serving endpoint without naming GLM-5.3.
That said, having the word "model" in the title is misleading imo
srdjanr 13 hours ago [-]
To me it's obvious that anything related to caching has little to do with the model itself. And it's nice to know from the title that you could only be affected if you used this model.
embedding-shape 8 hours ago [-]
It took me until "We have remediated the issue" to realize Cloudflare is talking about an issue they have in their infrastructure, up until that point they're not making that clear at all, and seems to point towards something else, unless you of course already are aware of what exact names Cloudflare uses for their products...
13 hours ago [-]
aliasxneo 19 hours ago [-]
It seems like you're trying to imply this was an intentional bias due to the nationality of the model authors. But the first sentence of the article is:
"On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026."
Meaning the issue was a cache exposure with the GLM-5.3 Workers AI model. If the title was, "Cache exposure issue with GLM-5.3," I would agree it's ambiguous and maybe intentionally harmful. But it literally says, "Workers AI model" which is clearly a Cloudflare product.
jacquesm 18 hours ago [-]
I did not mention anybody's nationality, nor did I imply that that was important.
Downthread I suggested "Cloudflare configuration error leaked data between user sessions" which I think is far better. You could improve on it further: "Cloudflare AI worker configuration error leaked data between user sessions". But this could have happened with any model.
weird-eye-issue 17 hours ago [-]
Why are you bringing up people's nationalities? This seems incredibly inappropriate
aliasxneo 8 hours ago [-]
Probably because there's an on-going conspiracy that's been all over HN that these large corporations are on a smear campaign against Chinese models.
20 hours ago [-]
kami23 23 hours ago [-]
I like this RCA a lot more than most. I appreciate the ownership and clear explanation of why their testing did not catch this bug.
Also impressed with the speed to address the issue unplug the model and then pretty quick remediation, the speed to remediation matters less to me, but good to see they can act quickly on their platform to kill the serving of the model.
segmondy 22 hours ago [-]
"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "
Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?
SillyUsername 15 hours ago [-]
So they knowingly deployed a configuration with an overflow risk, much like writing code with a buffer overflow exploit.
They assumed that it wasn't like an overflow exploit, despite the fact that's exactly what they'd set up to happen, on the grounds of thinking the equivalent of, "oh this probably uses smart pointers, we'll be fine".
That's how I read it anyway.
weird-eye-issue 21 hours ago [-]
Next sentence: "We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue."
jacquesm 20 hours ago [-]
Which is pointing the finger at Sglang, who also weren't responsible for this.
At a minimum someone should have realized during testing that their utilization figures are off by 40%.
A much better title would be "Cloudflare configuration error leaked data between user sessions"
weird-eye-issue 17 hours ago [-]
I mean they are responsible for this. It's definitely a bug on their end.
segmondy 20 hours ago [-]
Doesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.
weird-eye-issue 17 hours ago [-]
They thought it worked up to 1m token window
segmondy 16 hours ago [-]
why would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.
hedgehog 9 hours ago [-]
See my other comment, this is likely only for the prefill GPU pool, a normal 1M token session would work fine. I think Cloudflare should probably link an explainer on how model serving works at the top of the post because a lot of people are not familiar at that level of detail.
19 hours ago [-]
freakynit 20 hours ago [-]
Bro, I had to read this sentence in the context of the previous one, 3 fcking times.. and it sill does not make any sense to me.
Graziano_M 20 hours ago [-]
Sounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success.
freakynit 18 hours ago [-]
thank you...
lumzell 7 hours ago [-]
[flagged]
hedgehog 19 hours ago [-]
I read that as you can only do a single prefill up to 681k tokens, separate from cache hit or generation. I think people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare. If you run a normal coding session with many turns it would get to 1M ok.
embedding-shape 8 hours ago [-]
Shit happens, this is normal. However, this:
> We were aware of this mismatch before launch
Isn't normal. Am I misunderstanding or are they openly confessing they've stopped caring about quality at all? Why would you launch when you know it's wrong?
nijave 7 hours ago [-]
Maybe they forgot to run this one by the lawyers before posting. I'm surprised they'd publicly admit to that
adg001 18 hours ago [-]
Please consider to update the title in: Cross-request KV cache exposure in the AI model workers
thehamkercat 23 hours ago [-]
Other providers probably don't even know/care that this is happening
6thbit 23 hours ago [-]
Last year it happened to me more than once that dictating something in chatgpt would become a completely different message after processing, not in a misheard type of way but just a separate conversation more easily explained by the same type of KV cache issue here.
kami23 23 hours ago [-]
I think in the last year I've had it happen once where I seemingly got someone else's response, it was very odd. I rewound and got what I expected. It was correlated to Anthropic's roll out of new models at the time which always seems to cause instability in their apis.
kolanos 23 hours ago [-]
Other providers might have better isolated KV stores?
jmalicki 21 hours ago [-]
I've had this happen with ChatGPT too.
jmalicki 6 hours ago [-]
(Some random internet user pasted me things that were clearly from my ChatGPT session, he identified me because it had links to my github). So it wasn't a hallucination, it appeared to be some sort of KV cache leakage, though I obviously don't know the true root cause of the bug.
blarelewis741 8 hours ago [-]
[flagged]
Orygin 21 hours ago [-]
I wonder if the chatgpt shirtless guy is from the same kind of issue
alxsm 6 hours ago [-]
What worries me is what was actually in those leaked prompts. This is a million-token context model, and people use that context to paste whole repos and configs, .env files included. So the realistic worst case isn't someone's chat text, it's someone's API keys. The write-up explains the root cause well, but doesn't say whether the customers whose prompts ended up in other people's responses were notified.
yababa_y 8 hours ago [-]
Even cloudflare being unable to deploy this technology correctly makes me think there are some serious failures of knowledge transmission happening somewhere between theoreticians and practicians.
himata4113 6 hours ago [-]
cloudflare has never been known to deploy stable software especially for features early in their lifetime
armcat 8 hours ago [-]
A much better title for this would be something like "Cross-Tenant KV Cache Exposure in Shared LLM Inference: Analysis and Remediation". As others have pointed out there is no need to mention GLM-5.3 in the title. As someone that has been using GLM-5.3 I immediately thought there is a cause for concern here, only to find out there isn't.
ComputerGuru 8 hours ago [-]
Read this last night. I was completely thrown off when I read this line:
> We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue
Surely “we expected and error but got a successful response” merits investigating rather than moving on?
sauravadhk 17 hours ago [-]
It seems more specific to the platform than the model. Maybe Cross-request KV cache exposure in the AI model workers would be a better fitting title?
reassess_blind 19 hours ago [-]
Reminds me of a bug I found in a very popular browser testing service that spun up iOS vm's/devices for on-device testing.
I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other.
I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back.
All pretty disappointing.
18 hours ago [-]
8cvor6j844qw_d6 23 hours ago [-]
Interesting. That might explain the few times I've received completely unrelated responses from a certain provider's API.
nitishr 12 hours ago [-]
I'm just surprised more serverless providers haven't found (or publicly disclosed) similar issues, yet.
xmcqdpt2 11 hours ago [-]
Six months ago I had a very basic prompt to Claude at work return very specific, unrelated information that looked very much like it could have come from someone else in the company. We raised an incident and were told by Anthropic (in a 100% Claude generated email) that this wasn't possible and that AI sometimes makes mistakes like that. I'm not sure how much they investigated but apparently their email was enough for legal and compliance so eh.
0xbadcafebee 22 hours ago [-]
I kind of forgot about the shared tenancy attack model. I think I was assuming/hoping the software was magically isolating my tokens, when in reality that's impossible for shared tenants. There's whole classes of attacks on things like shared cache, before we get to the LLM-specific features and potential bugs. Enterprises with highly sensitive data should really be on non-shared hosts.
Alex13192 22 hours ago [-]
[flagged]
octoberfranklin 18 hours ago [-]
Upshot: even though sglang is written in Rust, they still got burned by undefined behavior from uninitialized memory.
himata4113 6 hours ago [-]
This is entirely self inflicted as SGLang is optimized for (near) maximum performance. Another issue is that this is dealing with a device driver rather than system memory which rust has very little control over, as in there is no native implementation in rust that allows you to address multiple-memory buffers. Think how old games had 16 different memory blocks all starting at 0 since the cpu could only address 65 kilobytes.
totetsu 20 hours ago [-]
Is anyone even offering single tenant inference as a service?
imhoguy 11 hours ago [-]
Still you may get traces of previous tenant if things are not cleaned up properly.
dools 15 hours ago [-]
That’s just renting GPU clusters
paaloeye 21 hours ago [-]
Is that a new class of tenant isolation vulnerability?
dnautics 20 hours ago [-]
Seems not? Just good old gpu allocations colliding
18 hours ago [-]
lianbuilds 20 hours ago [-]
[flagged]
yunohn 7 hours ago [-]
> We were aware of this mismatch before launch. We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue.
This is insane? They thought it would return errors for long context, which didn’t happen, and they proceeded assuming it was fine? Didn’t even investigate the known misconfiguration and issue before going to production?
brcmthrowaway 7 hours ago [-]
I should probably learn how inference is served at scale. Any have a zero to one reference for this?
This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
There's a huge amount of unnecessary extraneous details in the headline that distract from "this is a Cloudflare bug".
A better title would be "Cross-request KV cache exposure in Cloudflare provisioned GLM-5.3 Workers AI model" because it narrows culpability. Just posting it on the Cloudflare blog isn't enough for that in the age of AI and SEO data-scrapes and it'd be surprising to me if the authors didn't know that.
Did you ever given any thought about how confusing the title may be?
Did AI write this title and you just use it without ever wondering if it makes sense at all?
Which part of that is not claiming the exposure is in X model? Cloudflare is stopping to be cool, this is the start. I guess, cf is now big enough for enshitification to start.
Then you would used "endpoint" instead of "model". In fact, there's actually zero reasons to put any model or endpoint in the title if the bug is yours alone.
Most people will read this headline and assume Cloudflare is intentionally trying to squash open-weight models. Given how superfluous the model is to the content, it's easy to believe that narrative.
If that wasn't your intent, you really ought to change the title ASAP. Time is ticking before tech journalists and social media decide to run with that narrative and make your public relations department sad.
It effects the _endpoint serving GLM-5.3_
I'm not sure how else you'd identify the GLM-5.3 serving endpoint without naming GLM-5.3.
That said, having the word "model" in the title is misleading imo
"On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026."
Meaning the issue was a cache exposure with the GLM-5.3 Workers AI model. If the title was, "Cache exposure issue with GLM-5.3," I would agree it's ambiguous and maybe intentionally harmful. But it literally says, "Workers AI model" which is clearly a Cloudflare product.
Downthread I suggested "Cloudflare configuration error leaked data between user sessions" which I think is far better. You could improve on it further: "Cloudflare AI worker configuration error leaked data between user sessions". But this could have happened with any model.
Also impressed with the speed to address the issue unplug the model and then pretty quick remediation, the speed to remediation matters less to me, but good to see they can act quickly on their platform to kill the serving of the model.
Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?
They assumed that it wasn't like an overflow exploit, despite the fact that's exactly what they'd set up to happen, on the grounds of thinking the equivalent of, "oh this probably uses smart pointers, we'll be fine".
That's how I read it anyway.
At a minimum someone should have realized during testing that their utilization figures are off by 40%.
A much better title would be "Cloudflare configuration error leaked data between user sessions"
> We were aware of this mismatch before launch
Isn't normal. Am I misunderstanding or are they openly confessing they've stopped caring about quality at all? Why would you launch when you know it's wrong?
> We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue
Surely “we expected and error but got a successful response” merits investigating rather than moving on?
I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other.
I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back.
All pretty disappointing.
This is insane? They thought it would return errors for long context, which didn’t happen, and they proceeded assuming it was fine? Didn’t even investigate the known misconfiguration and issue before going to production?