Why the same question takes 150 seconds one time and 4 the next

The measurement is quickly told. 120 real documentation files, 149,597 tokens in total, go in as context to a Qwen3.8-27B on a DGX Spark, with a question behind them. 149 seconds pass before the first token of the answer. Then the same documents again, with a new question behind them: 4 seconds. And then that new question a third time, only now placed in front of the documents instead of behind them. 150 seconds.

Same corpus, same question, same card, same cache. A factor of 36 in between. The only thing that changed is the position of twelve words in the prompt.

What happens during those 149 seconds

Before a model can write the first token of an answer, it has to push the entire prompt through every layer once. That is the prefill. For every token, keys and values come out of every layer, and they land in the KV cache so that the decode phase afterwards doesn't have to start from scratch for each new token. How large that cache gets is covered in the piece on VRAM cost. This one is about time.

The prefill is the expensive phase on long contexts, because it is compute bound and the Spark only manages around 1,000 tokens per second on this model, which at 150,000 tokens comes to two and a half minutes during which the model emits nothing. The user sees a spinner.

A prefix cache keeps the result of that computation beyond the call, so that the keys and values are already sitting there when the next prompt arrives with the same beginning, and the engine only has to compute what is new at the end. vLLM has done this by itself since the V1 engine. No markers. Cloud providers sell the same mechanism as "prompt caching", with explicit markers and a discount on the token price.

The four scenarios

All four calls use the same corpus. The only difference is what sits in front of it and behind it.

(Model qwen3.8-27b in NVFP4 with an FP8 KV cache under vLLM, 262,144-token window, single requests. A second pass agrees to within two percent.)

Two things here deserve a second look.

B and C are equal. A new question behind a known corpus costs nothing measurable beyond the identical prompt. That is the real-world case: someone works with the same documents and asks the second, third, tenth question. Each one starts after 4 seconds.

And why 98.3 % rather than 100? Of 149,722 prompt tokens, 147,200 come from the cache. The remaining 2,500 or so are the tail behind the last fully cached block plus the new question, and those have to be computed in any case. At 1,000 tokens per second that is a good half of the 4 seconds. The rest is scheduling and the first decode token.

Why the question in front breaks everything

D is the case that makes no sense at first glance. The corpus was fully in the cache at that point. 147,200 tokens, block by block, already computed. Still 0 % hits.

How does the engine recognise a hit in the first place? Prefix caching matches from token 0, block by block, and the blocks are tied together by a hash chain. The hash of each block is formed from its content and the hash of all the blocks before it. As soon as one block differs, every following hash is a different one, even if its content is identical character for character. The question placed in front shifts the entire corpus into a different hash space. The cache is full. It just doesn't match.

This is not an edge case you have to construct. A timestamp in the system prompt does the same thing. "Today is 30 August 2026, 14:03" as the first line, and every call is cold again. A session name, a user ID, a "You are talking to Markus" right at the top: cold. Tool schemas that come out of a dict or a set and change their order between two calls: cold. And none of it shows, because the result looks identical. The service merely seems slow.

Why no hardware replaces this

Can't you just throw a faster card at it? Run the numbers once.

Cold, the prefill runs at 1,000 tokens per second, from the cache at 36,000. Anyone who wants the 4 seconds without a cache therefore needs a card that is 36 times faster at prefill than the Spark. It doesn't exist. The jump from a Spark to a datacenter card brings, in practice, a single-digit factor on prefill, and it costs a multiple. A factor of 36 is not something any card you can buy will give you.

The cache, on the other hand, costs nothing that isn't already there. It lives in the KV pool the engine has reserved anyway, and in vLLM it is a switch that is on by default. So you don't buy those 145 seconds with money. You buy them with the order of things in the prompt.

The reverse holds too: whoever has the order wrong can upgrade the card as much as they like and will never see the 4 seconds. They are optimising the wrong factor.

What this means for an agent

An agent is the best conceivable case for this cache. The history it sends along in full on every call is a growing prefix, because turn 10 begins exactly like turn 9 and only carries one tool result and one answer more at the end. System prompt, tool schemas, the first messages. All of it stays at the front and stays the same. As long as nothing variable slips in ahead of it, an agent pays the prefill for each turn only once.

From that follows a rule that is easy to keep in mind: static up front, variable at the end.

• Up front goes what stays the same between two calls. System prompt, documents, tool schemas, examples.
• At the end goes what changes. The question, the timestamp, the user context, the session ID.
• Sort stably whatever comes out of an unordered structure. Tools alphabetically, documents in a fixed order.

How do you tell that something is off? By the symptom "always equally slow". A service that takes just as long on the second and third question over the same documents as it did on the first probably has something variable too far forward. The model doesn't care. It computes dutifully. Only you wait.

One case is allowed and shouldn't alarm you. When an agent compacts its history because the window is filling up, it rewrites the prefix, and the next call is cold once. That is the price of compaction, and it is paid once. A timestamp up front is paid on every call.

What the measurement doesn't say

Measured at roughly 150,000 tokens, on one model, on one card, with single requests and no parallel load. Four things lie outside of that.

How long a prefix survives in the pool before other requests evict it is open, because the runs here were seconds apart, and in mixed operation with a pool of just under 970,000 tokens a prefix can disappear considerably earlier. How hits behave under several concurrent clients with different prefixes, likewise. Not measured. On small contexts the effect is necessarily smaller, because the prefill share is smaller. And the numbers hold for this model; the mechanism holds for vLLM in general, and in substance for the cloud APIs as well, which run their cache on the same prefix logic.

Anyone measuring this themselves will probably trip over two things. usage.prompt_tokens_details is always null on this vLLM image, so the cached_tokens known from OpenAI don't exist here, and the hit rate therefore has to come from the Prometheus counters under /metrics. And vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total count tokens, not blocks. The first version of the measurement script multiplied by the block size of 16 and reported 2.3 million "cached tokens" on a 150k prompt.

Once you've measured this, you read a prompt differently. The question is no longer just what's in it. The question is in what order.