feat: compile-cache executor + fastcache-cc sccache-style launcher - #33
Open
Yaraslaut wants to merge 23 commits into
Open
feat: compile-cache executor + fastcache-cc sccache-style launcher#33Yaraslaut wants to merge 23 commits into
Yaraslaut wants to merge 23 commits into
Conversation
Records the approved design consolidating the executor direction from docs/specs/executor.md into concrete decisions: server canonicalizes on store / client localizes on fetch (Option C), a new CompileCacheProtocol handler over the unchanged IStorage stack, client-tagged value regions (object blob never touched), a shared versioned canonicalization spec with a conformance suite, and executor-owned cohort prefetch over a Phase-1 Prefetch hook + cohort manifest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…canon v1) PathCanon maps absolute paths to machine-neutral <SRCROOT>/<BUILDTREE> tokens and back, and rewrites path spans inside captured compiler text regions (showIncludes / MSVC diagnostics) per a grammar tag. Governed by docs/specs/compile-cache-canon-v1.md with conformance vectors as the server<->client parity contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… regions) EncodeCompileValue/DecodeCompileValue serialize a compile result as an untouched object blob plus zero or more grammar-tagged text regions, with bounds-checked decoding that rejects truncated, wrong-version, or unknown-grammar frames. Big-endian lengths match the project wire helpers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ffects IStorage gains a Prefetch(key, now) primitive: warm an entry into the in-memory tier with no client-read side effect (no hit/miss stat change, no LRU-as-hit promotion). Default Peeks (correct for single-tier); LayeredStorage overrides to pull L2->L1 via the existing mirror path; Sharded/Tracing/Notifying/WriteErrorReporting forward it. Backs the compile-cache executor cohort prefetch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rage CohortManifest records which cache keys belong to a build cohort, stored in-cache under a control-byte-prefixed key so no new storage interface is needed. AddKey appends atomically via IStorage::Update (idempotent, capped at MaxKeysPerCohort); Keys lists them in insertion order. Backs the executor's leading-key cohort prefetch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, serve canonical on FETCH) Adds the CompileCacheHandler IProtocolHandler behind a new 0xFC autodetect magic byte. STORE canonicalizes every text region with the producer layout (object blob untouched), stores the machine-neutral value, and records cohort membership. FETCH serves the canonical value verbatim for the client to localize. Cross-depth portability verified end-to-end: a value stored from a deep layout localizes to a shallow consumer. CacheEngine gains Prefetch + Storage accessors the executor drives. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
STORE now records a reverse key->cohort index alongside the forward manifest. On a FETCH hit the handler discovers the fetched key's cohort and warms the rest of that cohort into L1 (after the reply is sent, guarded per connection so a cohort is primed once). Verified: fetching one member of a cohort warms its siblings into L1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reference-localizer client links the same PathCanon/CompileValue codec the server uses and speaks the 0xFC protocol over TCP: `store` compiles with /showIncludes and STOREs the framed result; `fetch` localizes the canonical value to the consumer layout and verifies every include path resolves. run-crossdepth.ps1 proves the value-portability guarantee with real cl.exe and clang-cl compilations — an object produced at a deep path is usable from a shallow checkout. Off by default (FASTCACHE_BUILD_TESTCLIENT). Prints generic counts only; no project source or private paths are read or emitted. Also lands the executor design spec that motivated the work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ze test path literals Adds CompileCache/ + CompileCacheHandler to the AGENT.md architecture map and an 'implemented (server side)' note to executor.md linking the design and canon spec. Replaces example checkout-path literals in the unit tests with a generic D:\project root so no specific checkout path is baked into fixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tocol fastcache-cc <compiler> <args> fronts a compile: computes a cross-machine key (preprocess output + checkout-relativized args + compiler id), FETCHes from fastcached, and on a HIT reproduces the object + replays stdout/stderr (showIncludes localized to this machine) so Ninja records valid deps. On a MISS it runs the real compiler with split stdout/stderr capture, STOREs the canonicalized result, and passes output through. Any cache error falls back to a real compile and warns (FASTCACHE_VERBOSE). Config via FASTCACHE_* env. Unit-tested: command parsing + key parity across checkout roots. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a HIT/MISS verbose trace to the launcher and an E2E harness that drives fastcache-cc as the compiler with real cl and clang-cl: first compile MISSes and stores, second (object deleted) HITs and reproduces the object byte-identically; content stored from a deep checkout HITs from a shallow one with showIncludes localized. All checks pass for both compilers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…real builds Validated against the real LASTRADA compile database (compile_commands.json): - Capture the preprocess for keying from stdout only (not merged 2>&1); the merged interleave of two buffered streams is unstable and was making the cache key nondeterministic. - Drain child stdout/stderr concurrently (a second thread) so large preprocessed output (>64KB pipe buffer) is not truncated/deadlocked. - Harden Windows argument quoting (double backslash runs before a closing quote) so MSVC/SDK include paths with spaces survive to the child compiler. - Skip translation units that reference __TIME__/__DATE__/__TIMESTAMP__ (they re-key every second and can never hit), matching sccache. Falls back to a plain compile. - Surface STORE outcome under FASTCACHE_VERBOSE for diagnosability. Sample of 150 real TUs: 99.3% hit rate on the second pass (147/148 cacheable; 2 .rc resource files fall back correctly, 1 residual header-introduced __TIME__ TU). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lue-cap guidance Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s-folder hits) RelativizeArgs only tokenized the source root, leaving include args that point into the in-checkout build tree (…\out\build\vcpkg_installed, generated headers) as machine-absolute paths. Those paths differ across checkouts, so the key differed and cross-machine/cross-folder hits collapsed to near zero on any real project. Now relativizes against both roots (PathCanon emits <BUILDTREE> for the longer-matching build-tree paths, <SRCROOT> otherwise). Caught by replaying a real 4223-TU compile database from a mirrored checkout at a different absolute path: cross-folder hit rate went 7.5% -> 100% on the verified sample after this fix. Unit test added for the build-tree case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The disk tier was hardcoded unbounded (opts.maxBytes = 0), so a build cache grew without limit. Add --storage-max-disk: the total budget is split evenly across physical shards and each CoW tree evicts its LRU tail to fit (the eviction path already existed and is tested). Default 0 keeps the previous unbounded behaviour. Lets an operator hold a cache within a fixed disk allotment (e.g. --max-memory 5G --storage-max-disk 10G). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compile the launcher from just its own sources plus the dependency-free PathCanon/CompileValue/Crc32c cores instead of linking the whole FastCache library, so it pulls in no vcpkg deps (zstd/lz4/yaml-cpp). Add FASTCACHE_STATIC_LAUNCHER (on by default) to statically link the MSVC CRT (/MT). A Release build then depends only on always-present Windows system DLLs (KERNEL32, WS2_32) — a 0.6 MB copy-anywhere executable needing no VC++ redistributable on build agents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage_max_value was already a YAML key; storage_max_disk (the on-disk L2 cap) was CLI-only. Wire it into YamlReader with the same byte-size grammar so the whole cache config can live in one file. Tests: both keys parse suffixes, disk defaults to 0 (unbounded), malformed value rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…yout LocalizeOne joined the root and tail with a hardcoded backslash while ToNative picked its separator behind #if defined(_WIN32). On POSIX the two disagreed and produced a mixed path (D:\project\include/foo.h), failing the Localize round-trip and cross-depth tests on every non-Windows runner. The separator is a property of the layout being localized to, not of the host: the cache is shared across machines, so a Windows consumer layout must localize to backslashes even when the daemon runs on Linux. Derive it from the target root (SeparatorOf) and use that one value for both the join and the tail (JoinLocalized); the #ifdef is gone, so output no longer depends on where the code was compiled. Also collapse LocalizeOne's two copy-pasted sentinel branches into a sentinel -> root descriptor table, and take RewriteRegion's callable by const& — it is invoked once per matched span, never forwarded, which clang-tidy flagged as cppcoreguidelines-missing-std-forward. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ne refs
clang-tidy flagged two defects in CompileCacheHandler.cpp that the previous
CI run never reached (it aborted at the first error in PathCanon.cpp):
- cppcoreguidelines-avoid-reference-coroutine-parameters: ReadLengthPrefixed
took ByteReader& . A reference parameter is not stored in the coroutine
frame, so it dangles if the caller's object dies across a suspend point.
Every other protocol coroutine in the tree already takes ByteReader*; this
was the lone deviation, so follow the established convention.
- readability-function-cognitive-complexity: Run scored 74 against the 60
threshold. Extract the STORE and FETCH arms into HandleStore/HandleFetch
returning a Next{Continue,Abort} outcome, pull the region loop into
CanonicalizeRegions and the warm-up into PrefetchCohort. Run is now a thin
dispatch loop over the opcode.
Behaviour is unchanged: same frames read, same replies written, same
best-effort cohort bookkeeping, same abort-on-write-failure semantics.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI's tidy job walks the translation units in order and stops at the first failure, so each fix uncovered the next layer. Sweeping every file this PR touches locally found 16 findings; this clears all of them in the PR's own files: - bugprone-unchecked-optional-access (6): the check cannot see a has_value() guard through Catch2's REQUIRE macro. Compare via value_or so the access needs no guard, keeping the assertion just as strong (an absent value fails the comparison instead of the REQUIRE). - readability-container-contains (4): find(...) != npos -> contains(...). - modernize-use-designated-initializers (3): brace-init of TextRegion. - cppcoreguidelines-avoid-reference-coroutine-parameters (1): WriteBytes took its payload by const&; a reference parameter is not stored in the coroutine frame, so take it by value. - bugprone-easily-swappable-parameters (1): StoreFrame took four adjacent string_views. Group them into a named StoreFields struct so the fields cannot be transposed at a call site. Left alone: a clang-analyzer-cplusplus.NewDelete report in SharedValue.hpp, which this PR does not touch and which the Linux/libc++ CI build does not flag (analyzer path-sensitivity differs by standard library). Behaviour is unchanged; 824 test cases / 11964 assertions pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compile-cache executor +
fastcache-cclauncherImplements the server-side executor from
docs/specs/executor.md§5 and adrop-in sccache-style compiler launcher that uses it — together making a
shared compile cache correct across machines at different checkout depths,
ending the
/showIncludespoisoning treadmill.The problem
A cached compile value is not a portable blob. On a hit, the producing
machine's captured
/showIncludesoutput is replayed verbatim; Ninja recordsthose header paths into
.ninja_deps. When the producer's build-tree→headerdepth differs from the consumer's (canonically, a CI runner nested one level
deeper), the replayed paths don't exist on the consumer, so Ninja marks
everything dirty and no-op builds recompile hundreds of objects. The existing
client-side repair is a last-writer-wins treadmill.
The approach (Option C — server canonicalizes, client localizes)
A new
IProtocolHandler(CompileCacheHandler, autodetect magic0xFC) overthe unchanged
IStoragestack:into machine-neutral
<SRCROOT>/<BUILDTREE>tokens, using the producer'slayout carried in the frame. The object blob is a separate field and is never
rewritten. The stored value has no machine identity — so no producer can
poison a shared entry.
to its own layout. The server stays off the per-consumer transform path.
The canonicalization rule is normative in
docs/specs/compile-cache-canon-v1.mdwith conformance vectors as the client↔server parity contract.
Server components
CompileCache/PathCanon— absolute↔token + showIncludes/MSVC-diagnostics/depfile region grammarCompileCache/CompileValue— object blob (untouched) + grammar-tagged text regionsCompileCache/CohortManifest— cohort-id→key-set + reverse indexIStorage::Prefetch— warm a tier with no read side effect (LayeredStorage pulls L2→L1; forwarded through all decorators)ProtocolAutodetect0xFCflavor +CompileCacheHandler(STORE/FETCH + leading-key cohort prefetch)The launcher —
tools/fastcache-ccA real drop-in for sccache, wired via
CMAKE_CXX_COMPILER_LAUNCHER:hash(compiler-id + preprocess output + checkout-relativized args)— same content at different checkout roots ⇒ same key, so machines share entries. Isolated inCacheKeyfor tuning./showIncludeslocalized to the local layout, so Ninja records valid deps.CreateProcess), STOREs the canonicalized result, passes output through.FASTCACHE_VERBOSE) warns. The build never breaks because the cache is down.FASTCACHE_ADDR/SRCROOT/BUILDTREE/COHORT/VERBOSEenv (theSCCACHE_*model).Testing
clangcl-debug,PEDANTIC_COMPILER_WERROR=ON), including unit tests for every server component, the launcher's command parsing and cache-key parity across checkout roots, and a poisoning-regression test.cl.exe+clang-cl):compile-cache-testclient/run-crossdepth.ps1— store from a deep path, fetch+localize from a shallow one; every include path resolves.ALL CROSS-DEPTH CHECKS PASSED.fastcache-cc/run-launcher-e2e.ps1— drives the launcher as the compiler: first compile MISSes and stores; second (object deleted) HITs and reproduces the object byte-identically; content stored from a deep checkout HITs from a shallow one with/showIncludeslocalized.ALL LAUNCHER E2E CHECKS PASSED.Whole-codebase validation (real compile database)
The launcher was exercised against a large real C++ codebase by replaying its
compile_commands.json(4223 TUs) throughfastcache-cctwice (populate, thenmeasure). Second-pass hit rate: 99.7% (393/394 cacheable) on a 400-TU
sample. This surfaced and fixed several real launcher defects (all now
covered by the fix commit):
2>&1folded diagnostics in at a non-stable interleave point, making the key
nondeterministic (nothing ever hit).
truncated large (>64 KB) preprocessed output nondeterministically.
MSVC/SDK include paths with spaces (
C:\Program Files (x86)\…) survive.__TIME__/__DATE__/__TIMESTAMP__are skipped (theyre-key every second; same as sccache).
Operational note: objects can be large (observed max ~122 MB), so run the
daemon with
--storage-max-value 256M(this also raises the frame-payload cap);the 16 MiB default would leave ~7% of this codebase's objects uncached.
Scope / follow-ups
/in some segments (Ninja matches separator-insensitively — cosmetic); non-showIncludesdiagnostic paths aren't localized yet./showIncludesdiff inexecutor.md§6 before freezing.Design record
docs/superpowers/specs/2026-07-23-compile-cache-executor-design.mddocs/superpowers/plans/2026-07-23-compile-cache-executor.md,…-fastcache-cc-launcher.mddocs/specs/compile-cache-canon-v1.md🤖 Generated with Claude Code