LLM context: sizing the window, KV cache and prompt caching

Size context for the task

A large window gives a model access to more material, but quality depends chiefly on relevance, order and the model’s ability to use that material, while every token consumes compute, memory or a billable input category.

• Too little context: the model lacks private, recent or specific facts and relies more heavily on its parameters.
• Useful context: the necessary instructions, sources and examples are present without excessive repetition.
• Too much context: distractors, contradictions and stale history can reduce quality and extend processing time.

Context optimisation covers four quantities that must be measured separately if complexity and cost conclusions are to remain meaningful.

• Prefill: parallel processing of the input tokens; standard dense attention has quadratic work in sequence length, while kernels, sparse attention and hardware change observed latency.
• Decode: sequential generation; with a KV cache, each new token generally reads a growing retained context without recomputing the whole prefix.
• KV storage: dynamic memory that is approximately linear in sequence length and batch size.
• API billing: a contractual schedule by token category, cache, storage, service tier or tool that does not necessarily mirror the provider’s physical resource cost.

When context becomes insufficient

An underspecified prompt does not automatically cause a hallucination, but it removes verifiable evidence and weakens tasks that depend on internal documents, transaction state or a complete chain of constraints.

• Long-tail knowledge: an internal procedure or rare reference may be absent from, or poorly represented in, model weights.
• Currency and traceability: retrieved documents allow an answer to be dated and cited rather than assumed current.
• Task coherence: dependencies, exceptions and previous decisions must fit within the state supplied to the model.

The relevant threshold is functional rather than lexical, so it should be found by adding and removing sources in an evaluation instead of classifying a fixed token count as optimal.

• Measure retrieval recall before blaming the context window.
• Confirm that decisive passages are present and internally consistent.
• Compare several context budgets with the same tests and versioned model.

What long-context studies show

Liu et al. observed an often U-shaped curve on retrieval tasks: depending on the model and test set, relevant information in the middle of a long prompt was used less successfully than the same information at the beginning or end [1].

• The study establishes positional bias in the experiments it covers, not a universal inability to use middle context.
• New models, lengths and tasks need to be retested within the target application.
• Placing stable instructions and priority passages in salient positions remains a design hypothesis to validate.

Several distinct mechanisms can weaken a long context, and separating them helps identify the appropriate correction.

• Attention sinks: some initial tokens receive disproportionate attention in the models studied by Xiao et al. [2].
• Semantic distraction: plausible but irrelevant details can derail a solution path, as tested in GSM-IC [3].
• Length alone: Du et al. report varying degradation across five models and several tasks despite perfect retrieval [4].
• Reasoning shift: a 2026 preprint observes shorter internal traces in certain long contexts, but this preliminary signal cannot be generalised to every model [5].

A bell-shaped reliability curve is a design intuition rather than a quantitative law: useful input can improve a task, a distractor can impair it, and the best balance varies.

• Build representative cases with explicit success criteria.
• Vary length, relevance, position and contradictions independently.
• Keep the smallest context that reaches the quality target with a stable margin.

Local KV cache: memory, prefill and decode

In local inference, weights are a relatively static load while the KV cache retains the keys and values produced for each processed token, allowing decode to reuse the prefix rather than recompute it at every step.

• Prefill computes the prompt representations and populates the cache.
• Decode reuses that cache but reads a growing history to produce each new token.
• Total memory also includes temporary activations, kernel workspaces, buffers and runtime fragmentation.

For a dense transformer in which every layer stores keys and values at the same precision, the following approximation exposes the architecture-specific parameters that determine memory use.

M_KV = 2 × L × H_kv × D_head × N_seq × B × P_bytes

• L is the number of attention layers.
• H_kv is the number of key/value heads; GQA or MQA reduces it relative to conventional multi-head attention.
• D_head is head dimension, N_seq retained sequence length and B batch size.
• P_bytes is bytes per element, such as 2 for FP16/BF16 or 1 for INT8; the factor 2 represents K and V.

Llama 3.1 8B provides a reproducible example with 32 layers, 8 KV heads and head dimension 128 [6], equivalent to 128 KiB of BF16 cache per token and sequence.

2 × 32 × 8 × 128 × 32,768 × 1 × 2 = 4,294,967,296 bytes = 4 GiB

• At 4,096 tokens, the same approximation gives 0.5 GiB.
• At 32,768 tokens, it gives 4 GiB, in addition to weights and engine allocations.
• At batch size 4 with equally long sequences, the raw cache rises to 16 GiB.

The formula is architecture- and runtime-specific because latent attention, sliding windows, paged caching, shared prefixes, quantisation and eviction can materially change the actual result.

Llama 3.1 8B scenario, BF16, batch 1	Theoretical weights	Calculated KV cache	Sum before runtime overhead
4,096 tokens	≈ 16 GB	0.5 GiB	≈ 16.5 GB
32,768 tokens	≈ 16 GB	4 GiB	≈ 20 GB

KV quantisation directly reduces the P_bytes term, but its impact on quality and throughput depends on model, engine and hardware, so a theoretical memory reduction is not a performance guarantee.

• INT8 halves raw storage relative to BF16; INT4 theoretically quarters it.
• KIVI studies asymmetric 2-bit per-channel keys and per-token values [10].
• PyramidKV allocates cache budgets by layer depth and token importance, which is compression rather than a uniform precision change [11].

Prompt caching: compare dated bills

Prompt caching lets an API recognise or retain a previously prefilled prefix, but a cheaper read does not imply zero provider resource use or guarantee reuse for every repeated request.

• A hit depends on a compatible prefix, routing, retention period and provider rules.
• Dynamic input and output tokens retain their own billing categories.
• Writes, storage duration or service tiers can add costs missing from a headline read price.

This comparison is frozen at 6 September 2026 and names the exact model and API, so it should be checked again before a purchasing decision because prices and thresholds change independently.

Provider and reference	Standard input	Cache read	Write / storage	Activation and documented threshold
Anthropic · Claude Sonnet 4.6, Claude API [12]	$3.00/M	$0.30/M	$3.75/M for 5 min; $6.00/M for 1 h	Automatic caching or cache_control breakpoints; 1,024-token minimum on 6 Sep 2026
OpenAI · GPT-5.5, historical model card dated 23 Apr 2026 [13]	$5.00/M	$0.50/M	No separate write line on this card	Automatic detection of supported prefixes from 1,024 tokens
Google · Gemini 3.1 Pro Preview, generateContent [14]	$2.00/M up to 200k	$0.20/M up to 200k	Explicit cache: $4.50/M token-hours	Implicit and explicit; documented 4,096-token minimum for this model, not a universal 32,768
DeepSeek · V4 Flash from 16 Aug 2026 [15]	$0.22/M off-peak; $0.44/M peak	$0.007/M off-peak; $0.014/M peak	No separate write line in the cited schedule	Automatic prefix caching; usage metrics must confirm the hit

With Google, the API determines which mechanism is available: Interactions only supports implicit caching, whereas generateContent can also create an explicit cache with a TTL and storage charge [14].

• Minimums are listed by model and may change.
• Implicit savings are passed through when a hit occurs, without a guarantee that a particular request will hit.
• Explicit caching guarantees the cached-content rate when referenced, but storage is charged by token count and duration.

A simple calculation shows why writes or storage belong in break-even analysis: for a stable 100,000-token prefix on Claude Sonnet 4.6 with a five-minute cache, the initial write costs $0.375 and each read $0.03 instead of $0.30 at the standard input rate.

• Two calls without caching cost $0.60 for the stable input.
• One write followed by one read costs $0.405, saving $0.195 in this example.
• The outcome changes if the second call arrives after expiry, the prefix changes or other pricing multipliers apply.

To improve hit rate without promising universal reuse, an application should stabilise the start of its prompt and move volatile values after that prefix.

• Put tools, system instructions, schemas and stable corpora before variable questions.
• Keep timestamps, identifiers and non-deterministic ordering out of the intended common prefix.
• Monitor cache creation, read and miss counters exposed by the API.
• Calculate total cost per useful result, including outputs and storage.

A measurable context pipeline

A robust architecture progressively reduces the material sent to the model while preserving necessary evidence, then verifies quality at each stage instead of assuming that one technique or token count suits every model.

• Retrieval: maximise initial recall against a versioned collection.
• Reranking: use a more precise model to order passages against the question.
• Assembly: deduplicate, resolve contradictions and place priority evidence in a tested order.
• Evaluation: measure accuracy, citations, latency, memory and billing across several budgets.

The LLMLingua tools illustrate three distinct compression trade-offs that need evaluation on the target documents and task.

• LLMLingua uses a smaller model’s perplexity and reports up to 20× compression in its experiments [16].
• LLMLingua-2 learns a question-agnostic keep/drop classifier and generally reports more moderate rates with faster processing [17].
• LongLLMLingua is question-aware and filters and reorders documents for long-context settings [18].

Multi-agent decomposition can reduce each agent’s context when subtasks are genuinely separable, but it adds messages, synthesis, inconsistency risk and billable tokens.

• Define structured outputs and acceptance criteria for each subtask.
• Pass the supporting evidence to the synthesising agent rather than conclusions alone.
• Compare the topology with one reranked call, because more agents guarantee neither lower cost nor higher quality.

A decision protocol

Context size should be treated as an experimental variable tied to a model version, inference architecture, cache policy and measurable business objective.

• Quality: test evidence coverage, correctness and resistance to distractors.
• Local performance: profile prefill time, decode throughput, weight VRAM and KV cache separately.
• API cost: use reported usage and dated rates, including writes, storage, outputs and tools.
• Operations: alert on falling hit rates, length drift and model changes.

Apply the method to your use case

The Junyr AI maturity audit is a free, no-commitment 30-minute video call that positions your organisation on the scale, identifies the main blocker and first sensible project, and supplies a one-page follow-up.

• Book the AI maturity audit.
• Explore pricing, budgets and routing in Token budgets and AI APIs: the FinOps guide for SMEs.

Sources

[1] Nelson F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024. https://arxiv.org/abs/2307.03172

[2] Guangxuan Xiao et al., Efficient Streaming Language Models with Attention Sinks, ICLR 2024. https://arxiv.org/abs/2309.17453

[3] Freda Shi et al., Large Language Models Can Be Easily Distracted by Irrelevant Context, ICML 2023. https://arxiv.org/abs/2302.00093

[4] Yufeng Du et al., Context Length Alone Hurts LLM Performance Despite Perfect Retrieval, Findings of EMNLP 2025. https://arxiv.org/abs/2510.05381

[5] Gleb Rodionov, Roman Garipov and George Yakushev, Reasoning Shift: How Context Silently Shortens LLM Reasoning, 2026 preprint. https://arxiv.org/abs/2604.01161

[6] Meta, Llama 3.1 8B Instruct configuration. https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/config.json

[10] Zirui Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. https://arxiv.org/abs/2402.02750

[11] Zefan Zhang et al., PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. https://arxiv.org/abs/2406.02069

[12] Anthropic, Prompt caching and pricing, accessed 6 September 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching

[13] OpenAI, GPT-5.5 model card and Prompt caching, model pricing dated 23 April 2026, accessed 6 September 2026. https://developers.openai.com/api/docs/models/gpt-5.5 and https://developers.openai.com/api/docs/guides/prompt-caching

[14] Google AI for Developers, generateContent and Interactions Context caching plus Gemini pricing, generateContent page updated 2 September 2026, accessed 6 September 2026. https://ai.google.dev/gemini-api/docs/generate-content/caching and https://ai.google.dev/gemini-api/docs/caching and https://ai.google.dev/gemini-api/docs/pricing

[15] DeepSeek, price change announced 13 August and effective 16 August 2026. https://api-docs.deepseek.com/news/news260813/

[16] Huiqiang Jiang et al., LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. https://arxiv.org/abs/2310.05736

[17] Zhuoshi Pan et al., LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. https://arxiv.org/abs/2403.12968

[18] Huiqiang Jiang et al., LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. https://arxiv.org/abs/2310.06839

Written by Paul-Antoine TUAL, AI Transformation Leader and creator of the Junyr Method™.