Summary
We have three context-reduction mechanisms and no measurement of what any of them cost in prompt-cache hit rate. Providers cache on a byte-identical prefix, so any technique that rewrites an earlier part of the prompt invalidates the cache from that point on. Some of ours rewrite the prefix once, which is a fine trade. At least one appears to rewrite it on every turn, which would be a straight loss: cached input tokens are billed at a fraction of fresh ones, so a technique that trims tokens while destroying the cache can cost more than sending the untrimmed prompt.
Nobody has measured it. This issue is to measure it and then fix or document what we find.
The three mechanisms
| mechanism |
what it rewrites |
how often |
truncateOlderMessages (agents/chat) |
old tool outputs to 500 chars, old text to 10,000 |
suspected every turn |
media eviction (@cloudflare/think) |
a file part becomes an [evicted …] marker |
once per message |
compaction overlays (agents/sessions) |
a span of turns becomes a summary |
once per span |
The specific suspicion
truncateOlderMessages keeps the most recent keepRecent messages (default 4) intact and truncates everything older. The boundary is computed from the length of the message list, so it slides forward as the conversation grows. On the turn after a message crosses that boundary, its full text is replaced by a truncated copy, in the middle of the prompt prefix. Every request after that has a different prefix from the one that was cached.
If that reading is right, a long conversation invalidates its cache on a regular cadence rather than converging on a stable prefix, and the cheapest fix may simply be to anchor the truncation boundary to something stable (a message id, or a compaction anchor) instead of an offset from the end.
Media eviction and compaction should both be one-shot: they rewrite the prefix once and the new prefix is then stable. That is the acceptable shape, and it is worth confirming rather than assuming.
Worth noting that freezeSystemPrompt already exists precisely to keep the system prompt byte-stable across turns, so the intent is established; the message prefix just never got the same treatment.
What to measure
Run a long multi-turn conversation against a caching provider and record, per turn, the cache-read and cache-write token counts from the model's usage metadata (usage.cachedInputTokens is already surfaced through Think's step context, and pi reports cacheRead / cacheWrite).
Do it four ways so each mechanism is isolated:
- everything off
truncateOlderMessages only- media eviction only (
mediaEviction: true, truncation disabled)
- compaction only
For each: cache hit rate per turn, total cached vs uncached input tokens over the run, and the turn index where the hit rate drops.
Acceptance
- A table of the four runs showing which mechanisms cost cache hits and how often.
- For any mechanism that invalidates on a repeating cadence, either a fix that stabilises the prefix, or a documented statement of the trade with the numbers behind it.
- Whatever we learn is written into the context docs, since today they explain what these knobs do but say nothing about their interaction with caching.
Related direction
These mechanisms currently live in Think, but they are not Think-specific. They are middleware over the message stream coming out of agents/sessions: they read history and shape it before it reaches a model. agents/context was just created as the home for prompt assembly, and that is where they belong, so any host on Sessions gets them rather than only Think.
The caching answer should inform that move, since a mechanism that has to anchor to a stable prefix needs to see the same reference points the context layer already tracks. Recommend doing the measurement first and the move second.
Summary
We have three context-reduction mechanisms and no measurement of what any of them cost in prompt-cache hit rate. Providers cache on a byte-identical prefix, so any technique that rewrites an earlier part of the prompt invalidates the cache from that point on. Some of ours rewrite the prefix once, which is a fine trade. At least one appears to rewrite it on every turn, which would be a straight loss: cached input tokens are billed at a fraction of fresh ones, so a technique that trims tokens while destroying the cache can cost more than sending the untrimmed prompt.
Nobody has measured it. This issue is to measure it and then fix or document what we find.
The three mechanisms
truncateOlderMessages(agents/chat)@cloudflare/think)[evicted …]markeragents/sessions)The specific suspicion
truncateOlderMessageskeeps the most recentkeepRecentmessages (default 4) intact and truncates everything older. The boundary is computed from the length of the message list, so it slides forward as the conversation grows. On the turn after a message crosses that boundary, its full text is replaced by a truncated copy, in the middle of the prompt prefix. Every request after that has a different prefix from the one that was cached.If that reading is right, a long conversation invalidates its cache on a regular cadence rather than converging on a stable prefix, and the cheapest fix may simply be to anchor the truncation boundary to something stable (a message id, or a compaction anchor) instead of an offset from the end.
Media eviction and compaction should both be one-shot: they rewrite the prefix once and the new prefix is then stable. That is the acceptable shape, and it is worth confirming rather than assuming.
Worth noting that
freezeSystemPromptalready exists precisely to keep the system prompt byte-stable across turns, so the intent is established; the message prefix just never got the same treatment.What to measure
Run a long multi-turn conversation against a caching provider and record, per turn, the cache-read and cache-write token counts from the model's usage metadata (
usage.cachedInputTokensis already surfaced through Think's step context, and pi reportscacheRead/cacheWrite).Do it four ways so each mechanism is isolated:
truncateOlderMessagesonlymediaEviction: true, truncation disabled)For each: cache hit rate per turn, total cached vs uncached input tokens over the run, and the turn index where the hit rate drops.
Acceptance
Related direction
These mechanisms currently live in Think, but they are not Think-specific. They are middleware over the message stream coming out of
agents/sessions: they read history and shape it before it reaches a model.agents/contextwas just created as the home for prompt assembly, and that is where they belong, so any host on Sessions gets them rather than only Think.The caching answer should inform that move, since a mechanism that has to anchor to a stable prefix needs to see the same reference points the context layer already tracks. Recommend doing the measurement first and the move second.