Added support for explicit cache markers - #179
Open
simonsoanes wants to merge 2 commits into
Open
Conversation
…ly what data needs to be cached and do not want the performance hit or eviction risk of caching their following RAG blocks (or want to have fallback to a specific point consistently). This is compatible with OpenAI/Azure/Dashscope explicit cache breakpoint support.
There was a problem hiding this comment.
🟡 Changes recommended
Part-level cache markers are currently collapsed to a single per-message marker (losing correct breakpoint placement), and the new scheduler/cache filtering lacks direct unit-test coverage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds end-to-end support for explicit prompt-cache breakpoints (via cache_control / prompt_cache_breakpoint) so clients can precisely bound what gets admitted into the prefix-cache index, reducing eviction risk from uncacheable or request-specific prompt segments.
Changes:
- Parse cache-control markers from OpenAI/Ollama tool declarations and OpenAI/Responses messages (including content parts).
- Track marker positions during prompt rendering via an invisible sentinel, producing explicit token-index breakpoints without changing the model-visible token stream.
- Use the last explicit breakpoint to restrict which full KV blocks are registered into the prefix-cache index.
File summaries
| File | Description |
|---|---|
| TensorSharp.Server/RequestParsers/ToolFunctionParser.cs | Threads wrapper-level JSON into function parsing to read tool cache markers. |
| TensorSharp.Server/RequestParsers/ChatMessageParser.cs | Extracts cache markers from messages and content parts for OpenAI + Responses shapes. |
| TensorSharp.Server/RequestParsers/CacheControlParser.cs | New shared parser for cache_control / prompt_cache_breakpoint marker shapes. |
| TensorSharp.Server/ChatGenerationPipeline.cs | Plumbs renderer-produced breakpoints into truncation and sequence creation. |
| TensorSharp.Runtime/Scheduling/SequenceState.cs | Carries explicit cache breakpoints and a precomputed “furthest breakpoint” limit. |
| TensorSharp.Runtime/Scheduling/ContinuousBatchScheduler.cs | Filters which full blocks are eligible for prefix-cache registration based on breakpoint limit. |
| TensorSharp.Runtime/OutputParser.cs | Adds CacheControl on tool definitions in the runtime model. |
| TensorSharp.Runtime/KVCachePromptRenderer.cs | Implements breakpoint sentinel injection + stripping while returning breakpoint indices. |
| TensorSharp.Runtime/ChatTemplate.cs | Adds CacheControl on ChatMessage to carry markers into rendering. |
| TensorSharp.Runtime/CacheControlMarker.cs | New runtime marker model for explicit cache-control hints. |
| InferenceWeb.Tests/KVCachePromptRendererTests.cs | Adds unit tests pinning breakpoint tracking behavior and token-stream identity. |
| docs/EXPLICIT_CACHE_MARKERS_DESIGN.md | Adds implementation design documentation for explicit caching. |
| .gitignore | Ignores large local model artifacts and IDE/user scratch files. |
Review details
- Files reviewed: 12/13 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+176
to
+180
| if (type == "text" && part.TryGetProperty("text", out var txt)) | ||
| { | ||
| textParts.Add(txt.GetString()); | ||
| if (CacheControlParser.TryParse(part, out var partMarker)) | ||
| msg.CacheControl = partMarker; |
Comment on lines
+296
to
+301
| @@ -289,6 +297,8 @@ public static List<ChatMessage> ParseResponsesInput(JsonElement inputEl, string | |||
| part.TryGetProperty("text", out var txt)) | |||
| { | |||
| textParts.Add(txt.GetString()); | |||
| if (CacheControlParser.TryParse(part, out var partMarker)) | |||
| msg.CacheControl = partMarker; | |||
Comment on lines
+550
to
+554
| // Unlike a placeholder, a breakpoint that cannot be found is skipped | ||
| // rather than fatal: a template is free to drop the content it was | ||
| // attached to (Gemma 4's strip_thinking filter does exactly that), | ||
| // and a cache hint is not worth failing a completion over. The cost | ||
| // of a dropped hint is a shorter cached prefix, never a wrong one. |
Comment on lines
42
to
+52
| BlockTable = new BlockTable(blockSize); | ||
| PromptTokens = new List<int>(promptTokens); | ||
| CacheBreakpoints = cacheBreakpoints; | ||
| if (cacheBreakpoints != null) | ||
| { | ||
| for (int i = 0; i < cacheBreakpoints.Count; i++) | ||
| { | ||
| if (cacheBreakpoints[i] > CacheBreakpointLimit) | ||
| CacheBreakpointLimit = cacheBreakpoints[i]; | ||
| } | ||
| } |
Comment on lines
+638
to
+643
| private bool IsBlockAllowedByExplicitMarkers(SequenceState seq, int blockIndex) | ||
| { | ||
| int limit = seq.CacheBreakpointLimit; | ||
| if (limit <= 0) return true; | ||
| return (blockIndex + 1) * _cfg.BlockSize <= limit; | ||
| } |
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core prompt-rendering and prefix-cache registration behavior (high-impact paths) and should get a final human review despite added tests.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
docs/EXPLICIT_CACHE_MARKERS_DESIGN.md:3
- The design doc refers to "TensorSharp2" in the title and intro, but this repository/project is named TensorSharp. This can confuse readers and makes the document harder to discover/search alongside the rest of the docs.
# Explicit Prompt Cache Markers - TensorSharp2 Implementation Design
This document outlines the design for implementing explicit prompt-cache markers in TensorSharp2, ensuring full compatibility with explicit caching specifications based on `cache_control` markers.
TensorSharp.Runtime/Scheduling/ContinuousBatchScheduler.cs:662
- Even when explicit cache breakpoints are set, this path hashes the full computed prefix and then filters blocks in the loop. Clamping the hashed prefix to the last block-aligned breakpoint avoids O(n) hashing work for blocks that can never be cached.
for (int b = 0; b < curFull && b < seq.BlockTable.Blocks.Count; b++)
{
var block = seq.BlockTable.Blocks[b];
if (block.ContentHash != null) continue;
if (!IsBlockAllowedByExplicitMarkers(seq, b)) continue;
- Files reviewed: 13/14 changed files
- Comments generated: 1
- Review effort level: Lite
Comment on lines
610
to
+612
| var block = seq.BlockTable.Blocks[b]; | ||
| if (block.ContentHash != null) continue; | ||
| if (!IsBlockAllowedByExplicitMarkers(seq, b)) continue; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is for clients that know exactly what data needs to be cached and do not want the performance hit or eviction risk of caching their following RAG blocks for example, or are batch processing uncacheable content over and over (or want to have fallback to a specific point consistently).
Note that using this without a precise reason will reduce the amount of cache hits, you need to have a special reason like the above to benefit from it.
This is compatible with OpenAI/Azure/Dashscope explicit cache breakpoint support.