How we cut Claude API costs by 10x by understanding prompt caching

BLOG (./blog/anthropic-prompt-caching-deep-dive.md)

How we cut Claude API costs by 10x by understanding prompt caching

May 15, 2026 · llm · anthropic · claude · caching · performance · infrastructure

Status: living document. Notes from an ongoing optimization track on a long-running local Claude Code fork. Numbers are real; redacted internal names where it didn’t matter.

TL;DR

Anthropic’s prompt cache is a 10× cost lever, and a minefield. Over the past six weeks we shipped a series of patches to a fork of the Claude Code CLI that turned a flaky 60% hit rate into a steady 97%. This post walks through each fix, why it mattered, and the diagnostics we built to find them.

If you run Claude in production and your cache hit rate isn’t north of 95%, one of these probably applies to you.

The setup

We run Claude as the brain of a personal AI assistant: long sessions, lots of tool use, multi-hour conversations, frequent background tasks. The default 5-minute cache TTL is too short for that pattern: go grab lunch and you pay a full cache write on your next message. At our context size (~200K tokens), a single cache miss on Opus is $3 to $8 for one turn that should have cost $0.30.

Doing this 30 times a day on the API would mean serious money. On a $200/month Claude Max plan, it means burning weekly usage budget against work that should have been free.

So: cache health became a measurable, daily SLO.

What “prompt caching” actually is

In one sentence: Anthropic caches a prefix of your request (tools + system + messages) and re-uses it on subsequent requests if the prefix is byte-identical.

Four things you need to know:

The breakpoint matters. You designate up to 4 cache breakpoints per request via cache_control: {type: "ephemeral"}. Anthropic caches everything up to the last breakpoint.
TTL is 5 minutes by default, 1 hour optional. The 1-hour TTL costs 2× the input price on write but reads cost the same ($0.50/MTok for Opus). Worth it for sessions with idle gaps.
Any change to the prefix invalidates everything. A single timestamp drift in a system event, one renamed tool, one re-stamped cache marker, and you pay full price for the whole prefix again.
Cache hits cost 10% of fresh input. Misses cost 125% (5m) or 200% (1h) of fresh input. So a miss costs roughly 12.5× a hit.

The diagnostics

You can’t fix what you can’t measure. Step one was building a cache observability stack.

1. Read the usage block directly

Every Claude response includes a usage block:

{
  "input": 6,
  "output": 122,
  "cacheRead": 207803,
  "cacheWrite": 205,
  "cost": { "total": 0.322 }
}

A healthy turn has input near zero and cacheRead carrying most of the load. A cold miss looks like input: 6, cacheRead: 0, cacheWrite: 235560, cost: 4.43. That’s a full $4 paid for what should have been $0.30.

We dropped a script (cache-status.sh) that parses the session JSONL log and prints last-N turns as a HIT/MISS table, plus session totals. It became the first thing we look at when costs feel off.

2. Dump the actual request payload

Reading usage tells you a miss happened, but not why. So we patched the Claude Code transport layer to dump the full Anthropic request envelope ({system, messages, tools, cache_control positions}) to disk on every request:

// [patch:payload-dump] capture the finalized payload for diagnosis
dumpAnthropicPayloadForDebug(payloadObj);

200 most-recent payloads ring-buffered to logs/<gateway>-payload-dumps/payload-<epoch-ms>.json. When a miss happened, we’d grab the two payloads (the last hit and the cold miss) and diff them byte-by-byte. Almost every cache regression we found, we found this way.

3. Classify misses by cause

Most cache misses fall into one of five buckets:

Session start: first turn ever. Unavoidable.
TTL expired: gap > 1 hour. Expected, not a bug.
Wake-injected: async exec/notification landed mid-conversation. The injection added a unique prefix that invalidated cache.
Content-induced: a large fetch/file read changed the message tail in a way that shifted cache breakpoints.
Breakpoint overflow: more than 4 cache_control markers in the request. Excess ones silently dropped.

We built cache-miss-diagnose.sh to read the session log + payload dumps, classify each miss, and tally cost per category. Single command, full forensic report.

The patches

This is the meat. Each fix targets one specific cache regression.

Patch 1: Enable 1-hour cache retention

Problem: Default 5-minute TTL meant every coffee break cost $3-8 on the next message.

Fix: Set PI_CACHE_RETENTION=long in the gateway env. This activates the cache_control: {type: "ephemeral", ttl: "1h"} path in the payload policy. Cache writes cost 2× but last 12× longer.

# ~/.<gateway>/config.json
env:
  PI_CACHE_RETENTION: long

Impact: Eliminated ~95% of “idle gap” cache misses overnight. The math works out as long as you re-use the cache more than 1.6× before it expires; for active sessions, easy.

Patch 2: Strip timestamp from system events

Problem: Async system notifications (heat reminders, completions, exec wakes) included a timestamp prefix: System: [2026-05-04 14:35:02] notification body.... Every system event injected a new unique string into the conversation prefix, busting cache prefix every single time.

Fix: Strip the per-event timestamp in session-system-events.ts:

-`System: [${formatTimestamp(event)}] ${event.body}` + `System: ${event.body}`;

The timestamp is still on the event metadata; just not in the rendered prompt text.

Impact: System-event-driven cache misses dropped from ~10/day to zero.

Patch 3: Bucket envelope timestamps to top-of-hour

Problem: Every inbound user message envelope ([Telegram from Mon 2026-05-04 12:35 GMT-4]) had minute-precision timestamps. Each message produced a unique header → cache miss.

Fix: Round to the hour in envelope.ts:

function formatEnvelopeTimestamp(d: Date): string {
  d.setMinutes(0, 0, 0);
  return formatLocal(d);
}

Messages within the same hour share cache prefix; cache refreshes naturally on the hour boundary.

Impact: Every Telegram message, every exec-approval followup, every tool re-prompt is now cache-eligible against neighboring turns.

Patch 4: Walk 3 most-recent user messages with cache_control

Problem: The original payload policy only stamped cache_control on the trailing user message. When a new user message arrived, the previously-trailing message LOST its breakpoint. Anthropic looks up cached prefixes by breakpoint position, so when the breakpoint set changed, the cached prefix key changed, and we paid a full cache write even though the content hadn’t changed.

Fix: Walk backward from the tail, tag the 3 most-recent user messages:

// [patch:cache-stability] Tag up to the 3 most-recent user
// messages with cache_control, walking backward from the tail. This
// keeps prior cache breakpoints stable as the conversation grows.
let userTagsApplied = 0;
const MAX_USER_BREAKPOINTS = 3;
for (let i = messages.length - 1; i >= 0 && userTagsApplied < MAX_USER_BREAKPOINTS; i--) {
  const message = messages[i];
  if (message?.role !== 'user') continue;
  if (tagUserMessageWithCacheControl(message, cacheControl)) {
    userTagsApplied += 1;
  }
}

When the conversation grows by one message, the previously-trailing user message KEEPS its cache_control (because we tag it again this turn), and Anthropic’s cached prefix key stays stable.

Impact: Mid-conversation cache misses cut by ~70%. This was the highest-leverage single fix.

Patch 5: Honor the cache boundary marker

Problem: Our system prompt has a stable upper section (identity, tool definitions, persona) and a dynamic lower section (live status, latest task data, today’s working notes). If you cache the whole thing, every dynamic-section change blows the cache. If you cache nothing, you pay full price every turn.

Fix: Insert an explicit marker () at the seam between stable and dynamic. The payload policy splits the system text at this marker, applies cache_control to the stable prefix block, and leaves the dynamic suffix uncached:

const split = splitSystemPromptCacheBoundary(record.text);
if (split.stablePrefix) {
  blocks.push({
    type: 'text',
    text: split.stablePrefix,
    cache_control: cacheControl, // ← cached
  });
}
if (split.dynamicSuffix) {
  blocks.push({
    type: 'text',
    text: split.dynamicSuffix,
    // ← no cache_control, free to change
  });
}

Impact: Daily memory updates, heat changes, and live status injection no longer invalidate the 30K-token static prompt above the boundary.

Patch 6: Route heartbeats to a separate cheap model

Problem: Background heartbeat checks (every 1-2 minutes, checking for tasks/completions/state) ran on Opus and pulled the full system prompt. Every poll re-cached 150K tokens at $0.20 a tick. Add it up: ~$10/day wasted on background polls touching the cached prefix.

Fix: Three config changes:

agents:
  list:
    main:
      heartbeat:
        model: zai/glm-5.1 # cheap model, separate cache
        isolatedSession: true # different session key
        lightContext: true # strip bootstrap injection

Heartbeats now use GLM-5.1 (flat $30/mo plan, unlimited), run in an isolated session, and skip the bootstrap entirely. Tiny tokens, no contamination of the main session’s cache.

Impact: Heartbeat cost dropped ~30×. Main session cache stays warm because heartbeats don’t touch it anymore.

Patch 7: Preserve cache when bootstrap regen is a no-op

Problem: Bootstrap context (the BOOTSTRAP.md that injects task counts, recent decisions, current state) regenerated on every session boot. Even when the content was unchanged, the regeneration rewrote the file with a new mtime, and downstream code treated it as a content change → cache invalidated.

Fix: Hash the regenerated content. If hash unchanged, don’t rewrite the file. mtime stays put, no cache invalidation downstream.

Impact: Session resumes that should be free actually are free.

Patch 8: Watch the cache-beta header, not the heartbeat

Problem: The anthropic-beta: prompt-caching-scope-2026-01-05 header is required for 1h TTL to actually persist. If an npm install or bundle rebuild drops it from the gateway, you silently lose cache retention and don’t notice for hours.

Fix: Move the integrity check from a heartbeat poll (which fires every minute and re-runs the same expensive check) to a launchd WatchPaths trigger on node_modules/@mariozechner + the bundle. Only fires on bundle/dep change. Dedup’d 24h alerts to Telegram on regression.

Impact: Caught two regression incidents within 30 seconds of them happening, before they’d ate any usage.

Patch 9: Fix Anthropic OAuth’s “out of extra usage” 400 on healthy Max plans

Problem (orthogonal but adjacent): Anthropic’s OAuth pipe classifies requests as “third-party app” traffic if the system prompt mentions multiple unknown tool names (session_search, skill_manage, web_extract, etc.). Such requests get routed to the extra-usage pool and 400 with "You're out of extra usage", even on a healthy Max subscription.

The classifier inspects:

System prompt text: flags requests with multiple unrecognized snake_case tool names
Tools array name fields: flags non-whitelist tool names
Missing metadata.user_id and x-anthropic-billing-header: bills against extra-usage by default

Fix: All gated on is_oauth=True:

Sanitize -native snake_case tool names in system prompt text via a substitution table (session_search → search, skill_manage → skills, web_extract → web_fetch, …)
Drop the tools array entirely on OAuth (Claude Code’s accepted tool vocabulary is small and doesn’t overlap with the ‘s natives)
Attach metadata.user_id from local identity
Attach extra_headers["x-anthropic-billing-header"] with version + fingerprint matching the Claude Code release identifier
Strip cache_control from system blocks the OAuth pipe rejects on managed surfaces
Wipe pre-existing per-message cache_control markers before stamping the OAuth marker (Anthropic enforces ordering: a ttl=1h block can’t follow a ttl=5m block)

Impact: Goal-lane tasks against Opus through OAuth started actually working. Token costs unchanged (the requests now route to the Max-plan pool instead of the extra-usage pool, where they belonged all along).

Patch 10 (in flight): Stop overflowing the 4-breakpoint limit

Problem: Anthropic enforces a hard max of 4 cache_control breakpoints per request. We were sending 7: 4 on the system blocks (billing header, Claude Code prelude, persona, main prompt) + 3 on user messages. Excess breakpoints silently dropped. When the breakpoint set changed turn-to-turn, the prefix-cache key shifted and we paid full price for an unchanged prefix.

Fix (this commit): In applyAnthropicCacheControlToSystem, only stamp cache_control on the LAST eligible system text block (or the block immediately before AGENT_CACHE_BOUNDARY if present). One system breakpoint + 3 user-message breakpoints = 4 total, within Anthropic’s limit.

- // stamps every eligible block
- for (const block of system) {
-   if (record.cache_control === undefined) {
-     record.cache_control = cacheControl;
-   }
- }
+ // stamps only the last eligible block
+ const lastIdx = findLastStampableBlockIndex(system);
+ if (lastIdx >= 0) {
+   system[lastIdx].cache_control = cacheControl;
+ }

Validation: payload diff before/after: 7 breakpoints → 4 breakpoints. cache-miss-diagnose tally over 24h post-deploy should show mid-conversation misses near zero.

What we measured

After the cache stability patches stabilized:

Hit rate: 60% → 97% on active sessions (200+ turns each)
Avg cost per turn: $0.80 → $0.32 on Opus 4.7
Mid-conversation cold misses: ~10/day → 0-2/day
Daily Max-plan budget usage: weekly token window goes from 20% leftover at end-of-week → 60%+ leftover

That’s not optimization, that’s a different cost regime. The biggest single learning: cache breakpoint position is as important as cache breakpoint presence. Half our regressions were “everything else looked fine, but the breakpoints walked off-track.”

Lessons we’d burn into onboarding

Cache observability is non-optional. Build the dashboards. Read the usage block on every turn. Diff payloads when a miss happens.
Never /model mid-session. Cache is per-model. Switching invalidates everything. If you need a different model, spawn a subagent in a new session.
Watch your system prompt prefix like a hawk. Any timestamp, any per-turn metadata, any “dynamic header” above the cache boundary kills you.
Tool definitions count. Adding/removing/renaming a tool invalidates. Tool arrays are part of the cached prefix.
Background polls deserve their own model. Don’t let cron jobs and heartbeats pollute the main session’s cache prefix.
The 4-breakpoint limit is real. Count your cache_control markers. The 5th onward gets dropped silently, and the silence is what bites.
Test in payload-space, not in usage-space. Dump full request envelopes. Diff them. Usage metrics tell you you have a problem; payload diffs tell you what it is.

What’s next

Pre-warming via max_tokens: 0 requests was on the table, but our 1h-TTL setup already keeps active sessions warm. The remaining cold misses (~$115/week aggregate) are entirely session starts: every fresh subagent or /new pays its first cache write. That’s intrinsic; no clever caching helps because each one has a different prompt.

The real next lever is fewer cold starts: reuse warm sessions, hoist common subagent setup into shared cached prefixes, push more work into persistent multi-turn sessions instead of one-shot spawns.

Or, eventually, Anthropic’s automatic-caching mode (single cache_control at the top level, the system auto-moves the breakpoint forward as conversation grows). We’d lose the ability to skip dynamic suffixes, but we’d gain bulletproof breakpoint management. Worth a benchmark.

Code references

All patches live in our fork of the Claude Code CLI on a long-running compat branch. They’re tagged with [patch:<area>] comments for easy grep:

grep -rh "patch:" src/ | grep -oE "patch:[a-z-]+" | sort -u
# patch:cache-debug
# patch:cache-stability
# patch:payload-dump

The diagnostics scripts (cache-status.sh, cache-miss-diagnose.sh, cache-misses.sh, cache-ttl.sh, cache-compare.sh, cache-turn.sh) live in our skills directory and are loaded automatically per agent context. They’re small, dependency-free, and rely only on jq + python3.

If we ever publish the diagnostic suite as a standalone tool, this post will get an update.

This is an opinionated post about a specific cost lever in a specific stack. Your mileage will vary if you’re on a different plan tier, different model, different prompt structure, or different traffic pattern. But the diagnostic methodology (usage block + payload dump + miss classification) works the same regardless.