One cache breakpoint cut my agent loop's input cost by 47%

An agent loop is not one API call. It is a sequence of them, and every iteration re-sends the entire conversation so far. That is not a design flaw — the model is stateless, so the conversation is the state — but it means cost grows with the square of the interesting bit rather than linearly.

Here is a real run from my MCP host. One twelve-word question, three iterations, eight tool definitions.

Look at what actually changes between rows: about 140 tokens. Most of those 7,542 tokens are the same bytes, three times.

One breakpoint

The system prompt and the tool definitions never change during a run. Only the conversation on the end of them grows. So put a single cache breakpoint on the system block:

system: [
  {
    type: "text",
    text: SYSTEM_PROMPT,
    cache_control: { type: "ephemeral" },
  },
],
tools: claudeTools,
messages,

One marker. Because the API renders the prompt as tools → system → messages, a breakpoint on the system block covers the tool definitions sitting in front of it too — which is where most of the weight is.

The same run, after:

The model still sees the same 7,540 input tokens. But a cache read costs roughly a tenth of a fresh token, so the full-price equivalent drops from 7,540 to about 4,004 — a 47% saving on input, from one line.

The break-even is two requests

A cache write costs about 1.25× a normal input token; a read costs about 0.1×. So caching pays for itself on the second request that hits the same prefix.

An agent loop makes three API calls in about ten seconds. This is not a marginal optimisation for high-traffic systems — it is free money in the specific shape of a tool-use loop, and it is the single highest-return change I have made to any of these projects.

Two ways to break it that produce no error at all

This is the part worth internalising, because both failures are silent. You do not get an exception. You get a bigger bill.

1. The prefix must be byte-identical

Interpolate anything variable into the cached region and every hit becomes a miss:

// Silently destroys every cache hit for the rest of time.
const SYSTEM_PROMPT = `You are a helpful assistant. Session: ${sessionId}.
Current time: ${new Date().toISOString()}.`;

A timestamp, a session id, a user name, a shuffled tool order — any of them. This is why in my loop the tool list is built once, outside the loop, and why the system prompt and the effort setting are constants rather than per-request strings.

2. input_tokens stops meaning what you think

After caching, input_tokens reports only the uncached remainder. Seeing it fall from 2,375 to 94 is not a measurement bug — it is the point. But if your cost dashboard reads input_tokens and nothing else, you will now under-report by a factor of twenty-five.

The real prompt size is:

const promptTokens =
  usage.input_tokens +
  usage.cache_creation_input_tokens +
  usage.cache_read_input_tokens;

And the real cost weights those three differently. If you are tracking spend, track all three fields separately from the moment you enable caching, not after your first surprising invoice.

One more thing that fails quietly

There is a minimum cacheable prefix — 1,024 tokens on the model I was using. Below that, the API silently declines to cache. No error, no warning, cache_creation_input_tokens: 0.

If your system prompt is short, the breakpoint does nothing. This is another reason to put the marker after the tool definitions rather than trying to cache a lean system prompt on its own: eight tool schemas comfortably clear the floor, where three sentences of instructions do not.

Where to put the breakpoint in a loop that grows

The general rule for a tool-use loop: cache the largest stable prefix, and put the breakpoint at its boundary.

For my host that is tools + system, because both are fixed for the run's duration. If your conversation has a long fixed preamble — a retrieved document set, a large few-shot block — that belongs inside the cached region too, provided it does not change between iterations.

What must stay outside is anything that grows: the message history itself. You can place additional breakpoints further down to cache a growing conversation incrementally, and for long-running agents that is worth doing. For a three-iteration loop, one breakpoint captured almost all of the available win, and the added complexity of managing multiple breakpoints was not worth the remainder.

The loop, with the reasoning in comments, is lib/agent-loop.ts. The live app streams cacheReadTokens into its trace panel, which is the easiest way to see this working on somebody else's numbers before you trust it on your own.