See what an LLM remembers while it generates.

The KV cache prevents every new token from recomputing the entire past. Build a prompt, decode token by token, and watch memory, latency, prefix reuse, and eviction change together.

Run the cacheUnderstand the math
KEYS STORE WHERE TO LOOK · VALUES STORE WHAT TO RETRIEVE · PREFILL BUILDS THE CACHE · DECODE APPENDS ONE POSITION · PREFIX REUSE SKIPS RECOMPUTATION · EVICTION TRADES MEMORY FOR CONTEXT · KEYS STORE WHERE TO LOOK · VALUES STORE WHAT TO RETRIEVE · PREFILL BUILDS THE CACHE · DECODE APPENDS ONE POSITION ·
512cached tokens
64 MBKV memory
128 msprefill saved
0%prefix hit rate
KeyValueEvicted

The cache turns quadratic repetition into incremental work.

Prefill

All prompt positions are processed in parallel. Their projected keys and values are stored once for every layer.

cache bytes = 2 × L × Hkv × D × T × bytes

Decode

Each generated token adds one key and one value per layer. Attention reads the existing cache instead of recreating it.

growth/token = 2 × L × Hkv × D × bytes

Grouped-query attention

Many query heads share fewer KV heads. Reducing KV heads from 32 to 8 cuts cache memory by 4× without changing query-head count.

MHA Hkv = Hq · GQA Hkv < Hq

Read the 3D rack.

Rows are transformer layers

Every cached token contributes a key and value at every layer. The visual groups layers into rack shelves; longer trails mean more cached positions and more GPU memory.

Horizontal motion is decoding

New positions enter at the right. Under a sliding window, old positions leave at the left. Sink-token mode preserves a small beginning region while evicting the middle.

Brightness is reusable work

Prefix-matched positions glow. A server can reuse their KV tensors, avoiding repeat prefill compute. Reuse requires an identical prefix and compatible model configuration.

Operational checklist

Measure per-token growth

Calculate cache bytes before choosing concurrency. A model that fits alone may run out of memory when many sequences decode together.

Keep prefixes stable

Put reusable system instructions and shared documents first. Volatile timestamps or user IDs near the beginning destroy prefix-cache hits.

Test quality after eviction

Sliding windows save memory but remove evidence. Long-context evaluation must include recall from early, middle, and recent positions.

Why are both keys and values cached?

During attention, the new query scores cached keys to find relevant positions, then combines the corresponding cached values. Both are required.

Does the cache store model knowledge?

No. Model knowledge lives in weights. KV cache entries are request-specific activations representing the current sequence.

Why can prefix caching reduce time to first token?

If a matching prefix was already processed, the server can load its stored keys and values and prefill only the unmatched suffix.

Primary sources

Grouped-query attention

Ainslie et al., GQA studies grouped KV heads between multi-head and multi-query attention.