Why your 8 GB model needs 20 GB of memory

You download a model, see a number next to the filename and do the arithmetic with it. The Q8_0 file of Qwen3-VL-8B-Instruct is 8.5 GB - so it fits onto a 24 GB card with plenty of room to spare. And the server does start, does answer, all fine. Then a second user shows up, someone drops in a longer document, and suddenly the log reads CUDA out of memory.

The file size was never the memory requirement. It is just the one item that stays put.

Three items, and only one of them is constant

What an inference engine occupies on the card is three things:

• The weights. That's the file. 8.5 GB on disk are 7.9 GB in memory - disks count decimal, drivers count binary. This number never changes, whatever happens.
• The overhead. CUDA context, working buffers, allocator fragmentation. Just under a gigabyte plus a few percent on the rest. Annoying, but manageable.
• The KV cache. The item nobody has on their list - and the only one that grows while the server is running.

Live experimentcolumns = tokens · bands = layers

Cache matrix

Every column of the matrix is one token, every band a group of layers. Whatever has been computed once stays put and is only read from then on. Prefill fills all prompt columns in a single pass; after that each decode step appends exactly one column - the counters below keep track.

ExampleShort questionInfrastructure question

Prompt as tokens

1. You
2. ·are
3. ·a
4. ·ters
5. e
6. ·assistant
7. .
8. Why
9. ·is
10. ·the
11. ·first
12. ·token
13. ·slow
14. ?

14 prompt tokens, 0 generated · dots stand for spaces · the split is a fixed example, not a live tokenizer

K/V matrix

0 of 24 columns filled

5101520

• just computed
• cached, read only
• carried over from the previous request
• still empty

The 6 bands stand in for the 32 layers of the reference model - each single one holds its own key-value pair per token.

Prefill · 14 tokensNext tokenno cacheSecond requestReset

With cache: prefill computes the prompt once, every further token appends one column. Switching example or mode starts a fresh session.

computed

0

columns since prefill

reused

0

columns read instead of computed

cache in use

0 KiB

128 KiB per column

Work for this request

with cache (linear)

0 = 14 prefill + 0 × 1

without cache (quadratic)

0 · last step alone: 14

The two numbers are not estimates against each other: 0 computed plus 0 reused columns are exactly the 0 columns of the no-cache case. The cache does not invent work, it puts it away - here by a factor of 1.

Memory

Cache right now0 KiB · 0 columns

Full context window1.0 GiB · 4.2 % of a 24 GB card

Context length8,192 tokens

Reference model: 32 layers × 8 KV heads × 128 dimensions × 2 (keys and values) × 2 bytes (FP16) = 128 KiB per token. The lower bar measures against 24 GiB of graphics memory.

ReadingNothing computed yet. The prompt is there, the matrix is empty - exactly the state in which a request reaches the model.

Guided steps

Five clicks, five observations. The button sets the lab up accordingly; the insight opens once what it shows has been on screen.

1. 1Start the prefill and watch the matrix.
   Start prefill
   Show insight
2. 2Generate five tokens one at a time - hit “Next token” five times.
   Prepare for decode
   Show insight
3. 3Switch the cache off and generate five tokens again. Compare the counters.
   Cache off & prefill
   Show insight
4. 4Drag the context slider to its longest setting and watch the second memory bar.
   Context to long
   Show insight
5. 5Send the same session a second time: same opening, new question.
   Send second request
   Show insight

What usually gets misunderstood here

The model re-reads the whole text for every token.

→Not the text - its representations are already in the matrix. Exactly one column is computed anew, everything before it is read. The no-cache switch shows what the other reading would cost.

The KV cache is an answer cache.

→It stores intermediate states of a running generation, not results. The same question twice produces two generations - the cache saves the prefill, not the answering.

The context window is just a software limit.

→It has a physical side: every token in the window takes its place in this matrix. The lower bar shows how much graphics memory a full window needs for that alone.

The cache isn't a luxury you could switch off. Without it, the model would have to recompute attention over the entire text so far for every single new token. The cache holds the keys and values of all previous tokens, so each decode step only adds one column instead of redoing everything. It is the reason the answer trickles out evenly after the initial pause instead of getting slower and slower. You pay for it in memory.

The formula, on a real model

Per layer and per token, a model stores:

2 (keys and values) × num_key_value_heads × head_dim × bytes per element

For Qwen3-VL-8B-Instruct the values sit in the repo's config.json: 36 layers, 32 attention heads, but only 8 KV heads, head_dim 128. The KV cache is usually kept in FP16, so 2 bytes per element:

2 × 8 × 128 × 2 bytes  =  4,096 bytes per layer and token
× 36 layers            =  147,456 bytes  =  144 KiB per token

144 KiB sounds like nothing. The point is what that number gets multiplied by: the context length and the number of concurrent requests. Neither knob belongs to the model. They belong to how it is used.

(Weights of 7.9 GB and overhead are included in the last column.)

Three users at 24k context each - a longer document plus conversation history, nothing unusual in an agent's day - and the 8 GB file has turned into 20 GB of memory. At that point the cache is larger than the model. At the full context window the arithmetic stops working altogether: the 262,144 tokens printed on the model card cost 36 GB for a single request - in cache alone.

And the last two rows are identical on purpose. The cache doesn't care whether one user occupies 128k or four users occupy 32k each. It only knows the product.

What follows from this - three things that otherwise look unrelated

Why GQA was invented. The formula says num_key_value_heads, not num_attention_heads. On this model that's 8 instead of 32: four attention heads share one set of keys and values. That is grouped-query attention, and the effect is exactly the factor of 4. Without GQA - that is, with classic multi-head attention - the same cache would sit at 576 KiB per token, and the row "32k context, two requests" would read 36 GB instead of 9. GQA isn't a quality improvement. It is the answer to this table.

Why vLLM manages the cache in pages. At startup an engine doesn't know how long the answers will get. The naive route is to reserve room for the full context window per request - and then 36 GB sit occupied while 2,000 tokens are actually in there. PagedAttention instead breaks the cache into fixed-size blocks and hands them out like an operating system hands out memory pages: nothing is reserved until a block is needed, and identical prefixes across requests share the same blocks. That's why the same card serves a multiple of the requests under vLLM that an engine with rigid reservation manages.

Why a server that ran for one user falls over at five. The weights sit there once. The cache sits there per request. A server standing at 13.9 GB with one user appears to have 10 GB of room left on a 24 GB card - in reality that's enough for exactly two more requests of that size. Memory scales linearly with concurrency; the feeling that "it runs fine" does not.

What you do about it

Trimming context is the strongest lever. Not the model's context window, but what actually goes into it. Every token you keep out of the history saves 144 KiB - times requests, times runtime.

Cap the window hard. In vLLM that's --max-model-len. It limits how much context a single request may occupy and turns an unknown upper bound into a known one. Without it the engine plans for the model card's 262,144 tokens - and won't even start on a 24 GB card. Its companion is --max-num-seqs: the second knob from the table, capped as well.

Choose the architecture, not just the parameter count. Models with a limited attention window only keep part of their layers at full length. In gemma-4-12B-it, 40 of 48 layers are capped at 1,024 tokens; their cache stops growing there. The result: under the load "32k context, two requests" the 12B model needs 18.3 GB - slightly less than the 18.7 GB of the 8B model next to it. At 128k context it's 31.0 versus 47.3 GB. The smaller file is not automatically the smaller server.

Quantise the cache - with reservations. FP8 instead of FP16 halves the item exactly; 9.0 GB become 4.5. That's the last reserve, not the first move: a quantised KV cache costs quality much faster than quantised weights do, because every error in it propagates into all following tokens.

Once you've done this arithmetic, you read model cards differently. The file size tells you whether the model fits on the card. It doesn't tell you whether the server does.