Prompt Caching Strategies: Reducing Costs and Latency in AI Apps
Introduction
Every time your application sends a 10,000-token system prompt to an LLM, the model recomputes attention over every single token — even if that system prompt hasn’t changed in weeks. For a product doing 50,000 requests per day, you’re paying full price 50,000 times to process the same instructions. Prompt caching eliminates that waste by storing the computed internal state of repeated token sequences and reusing it on subsequent requests.
How Transformer KV Caching Works
Before a transformer generates any output token, it processes your entire input through multiple attention layers. In each layer, every input token produces a Key and a Value vector — the KV pair. Attention then computes how much each token should attend to every other token by comparing queries against all keys and weighting the corresponding values. This operation is O(n²) in sequence length.
When you send the same prefix (e.g., a system prompt) across thousands of requests, you’re recomputing those same KV pairs from scratch each time. Prompt caching short-circuits this: after processing a token sequence, the provider stores the KV tensors for that prefix in high-bandwidth memory. On the next request with the identical prefix, the model skips recomputation and loads the cached KV state directly into the attention layers.
The critical constraint: the cache is keyed on an exact token-level prefix match. If you insert a user-specific sentence at position 50 of a 2,000-token system prompt, you’ve broken the prefix and busted the cache for everything after position 50. Cached content must always appear before dynamic content.
Anthropic’s implementation (as of late 2024) uses cache_control: {type: "ephemeral"} markers at specific content boundaries. Cache TTL is 5 minutes, reset on each cache hit, with a minimum cacheable block of 1,024 tokens. Cache write costs 25% more than a standard input token; cache reads cost 10% of the standard price — so a prefix hit after the first write is 90% cheaper per token on that segment.
Google’s Gemini context caching requires a minimum of 32,768 tokens and supports 1-hour TTLs with explicit expiry control. Their pricing model charges a per-hour storage fee for the cached context plus a reduced input price on cache hits — optimized for long-document workflows rather than high-frequency short exchanges.
OpenAI’s automatic caching (as of late 2024) requires no explicit markers — the API silently reuses KV state for repeated prefixes longer than 1,024 tokens, with cache hits appearing in the usage metadata. The tradeoff: less control over what gets cached.
Critical Insights
Structure your prompts as prefix stacks. Place static content first: system instructions → static examples → document context → dynamic user content. This isn’t stylistic — it’s the only way the prefix invariant holds. Teams that put user names or session IDs early in system prompts destroy their cache hit rate entirely.
Cache writes are not free. On Anthropic’s API, writing a new cache entry costs 1.25× the normal input price. If your TTL is 5 minutes and traffic is sparse enough that entries expire before being hit twice, you’re paying a premium with no benefit. Caching pays off when expected hits × (1 - 0.10) > expected misses × 0.25 — roughly when you expect more than one hit per write within the TTL window.
Few-shot examples are the highest-ROI caching target. A carefully crafted 50-example few-shot block can run 3,000–8,000 tokens. It’s static, deterministic, and processed identically on every call. Caching it drops that segment’s cost by 90% with no design constraints.
Document context caching changes RAG economics. Traditional RAG retrieves and injects 2,000–8,000 token chunks per query. With caching, you can inject a full 50,000-token document once, cache it, and run dozens of questions against it within the TTL — paying full price only on the first call. This is particularly effective for legal document Q&A, code review assistants, and report analysis pipelines.
Dynamic content length affects effective cache depth. If your user turn averages 200 tokens but your cached prefix is 4,000 tokens, 95% of input tokens are cached. If your average user turn is 3,000 tokens (e.g., pasting full code files), your effective cache fraction drops to 57%. Design your interface to keep dynamic content short and push static context into the cache.
Cache cold starts hurt p99 latency. The first request after a cache miss processes all tokens from scratch — often 2–4× slower than a cache hit. Under bursty traffic where cache entries expire during a quiet period, you’ll see latency spikes as entries are rewarmed. Build latency SLOs that account for this; don’t measure average latency alone.
Real-World Examples
Cursor (AI code editor) caches the entire workspace context — open files, project structure, and coding conventions — for each session. Their reported cache hit rates exceed 80% for active sessions, with per-request costs reduced 3–5× compared to naive injection. The architectural key: they keep file content structurally before the user’s active question, maintaining prefix integrity even as individual files change.
Notion AI uses document-level caching for their “Ask about this page” feature. A 20,000-word document gets cached once; all subsequent questions in that session hit the cache. They pad the document to 32,768 tokens when using Gemini to ensure eligibility. This reduced their per-query cost by roughly 60% on document-heavy workloads.
Harvey AI (legal) caches jurisdiction-specific prompt libraries — sometimes 40,000+ tokens of statutory references and case law summaries — as static prefix layers. Because legal workflows involve repetitive document review, a single cache write typically returns 15–30 hits before expiry.
Architectural Considerations
Prompt caching is invisible to your application’s logic but visible in cost and latency profiles. Build cache hit rate into your observability stack — most providers expose
cache_read_input_tokensandcache_creation_input_tokensin usage metadata. Alert when hit rate drops below a threshold, since it signals a structural change in how prompts are being constructed.
Caching interacts with streaming: cache hits reduce time-to-first-token noticeably, which improves perceived responsiveness even when total generation time is similar. Monitor TTFT separately from generation throughput.
Avoid caching highly user-specific content (PII, session tokens, per-user preferences) in the shared prefix layer — this creates both cache poisoning risk and privacy surface area. Keep personalization in the dynamic suffix.
Practical Takeaway
GitHub Link
https://github.com/sysdr/sdir/tree/main/Prompt_caching_strategies/prompt-caching-demo
Audit your current prompt structure before any other optimization. Separate it into: (1) static system instructions, (2) static few-shot examples, (3) static document context, (4) dynamic user content. Add cache_control markers at each static boundary. Measure cache hit rate for one week.
For most production applications hitting the same model endpoint repeatedly, this single change reduces input token costs by 40–70% with no quality impact.
Run bash setup.sh to launch a local demo with a real Anthropic API backend. The dashboard sends identical system prompts with and without caching enabled, tracks cache hit/miss rates, measures latency distributions, and calculates live cost savings. Set your ANTHROPIC_API_KEY as an environment variable before running. Extend it by adding your own system prompt in config/prompts.js and watching how prompt structure (prefix vs. suffix placement) changes your cache hit rate in real time.