feat(experiment-runner): add deterministic trial engine with immutable evaluator and journal replay - #11
Open
undeemed wants to merge 16 commits into
Open
feat(experiment-runner): add deterministic trial engine with immutable evaluator and journal replay#11undeemed wants to merge 16 commits into
undeemed wants to merge 16 commits into
Conversation
…aluator and journal replay
Add the experiment engine phase: typed experiment contracts, a pure immutable
evaluator, a deterministic measurement model, and a trial runner that measures
baseline and candidate, gates them through the evaluator, and journals each
trial as a self-describing record that replays and re-evaluates from the
journal alone.
- contracts: experiment spec, decision bounds, metric sample/summary, and
verdict wire types, with schemas/{experiment,verdict}.schema.json kept in
sync (contract tests enforce field and enum parity).
- control-plane: additive read/journal APIs (snapshot, record_trial,
read_trial, trial_ids) over a new experiment_trials table; no change to the
existing journal schema or lifecycle.
- experiment-runner: immutable evaluator, deterministic model, trial runner,
and replay; integration tests for a promoted trial, a rejected-and-preserved
trial, journal-only replay reproducing the verdict, and the MVP acceptance
criterion.
Safe-alpha limitation: mock capabilities are leased and the broker lifecycle
always rolls back, so the verdict gates whether the candidate is applied at
all rather than whether it persists; durable keep-or-rollback awaits the
privileged broker.
…, nested schema parity
…ghten replay gate
…, metric-sample schema
…als, and replayed bounds
…ne and modeled capability
…report rejection reason
…fix baseline reason
…t-record lifecycle
…y docs, report lost record
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Implement the experiment-engine phase of fpsmaxxing: baseline/candidate trials plus a decision gate, per docs/IMPLEMENTATION_PLAN.md. Deliverables: (1) a typed experiment spec (hypothesis text, target capability ChangeRequest, warmup, repeated baseline/candidate measurement counts, and correctness/constraint bounds), serde+schemars-typed in crates/contracts with schemas/{experiment,verdict}.schema.json kept in sync (contract tests enforce field/enum parity); (2) a deterministic, immutable evaluator - a pure function of recorded baseline+candidate samples and fixed DecisionBounds, no LLM and no wall-clock, applying an ordered fixed-threshold rule (minimum sample count, then temperature/power/error ceilings, then mean-FPS improvement threshold), unit-tested with known-answer fixtures; (3) a trial runner in apps/experiment-runner that measures a baseline, measures the candidate, and runs the candidate through ControlPlane::run_lifecycle gated by the evaluator verdict, journaling a self-describing TrialRecord so the trial replays and re-evaluates from the journal alone without chat history; (4) integration tests against sidecars/mock-provider for a promoted trial, a rejected trial that leaves the baseline untouched, and a journal-only replay reproducing the identical verdict; (5) the MVP acceptance criterion 'one measured experiment is promoted or rejected using the immutable evaluator', demonstrated in a test.
Key decisions a diff-only reviewer would miss: Control-plane changes were deliberately kept additive and minimal per the task constraint - a new experiment_trials table plus read/journal APIs (snapshot, record_trial, read_trial, trial_ids); the existing journal schema and broker lifecycle were NOT restructured. Safe-alpha limitation (intentional and documented in runner.rs and the commit body): mock capabilities are leased and ControlPlane::run_lifecycle always restores the pre-state before returning, so the verdict gates whether the candidate is applied at all (Promote exercises the full snapshot/preview/apply/verify/rollback lifecycle; Reject never mutates the knob) rather than driving physical persistence; durable keep-or-rollback awaits the privileged broker. The measurement model is a deterministic stand-in for PresentMon/hardware telemetry, which is unavailable on this Linux aarch64 safe-alpha path - it is a pure function of the knob value so trials replay bit-identically. No heavyweight statistics dependencies were added (deny.toml gates deps); the decision rule is intentionally a simple documented fixed-threshold comparison of means with a minimum sample count. Linux-only, mock provider is the provider. A sibling task (fpsm-wdog-vv) is concurrently adding watchdog journal-read logic on a separate branch; my control-plane additions are purely additive to avoid conflict.
What Changed
apps/experiment-runneras a library plus demo binary: a deterministic measurement model standing in for live telemetry, a pureevaluatefunction applying an ordered fixed-threshold rule (minimum sample count, then temperature/power/error ceilings, then mean-FPS improvement) with no clock or LLM input, and a trial runner that measures a baseline and a candidate, runsControlPlane::run_lifecycleonly on aPromoteverdict, and appends a versioned self-describingTrialRecordcarrying the spec, samples, verdict, and the lifecycle outcome or broker error.replay_trialre-evaluates a record from the journal alone and re-applies the full policy gate, reporting whether the row is one this runner could have written.crates/contracts(ExperimentSpec,DecisionBounds,MetricSample,MetricSummary,Decision,VerdictReason,Verdict) with policy-owned bound constants (MAX_LEASE_SECONDS,MAX_SAMPLES,MIN/MAX_HYPOTHESIS_CHARS, and the decision ceilings), and newschemas/{experiment,verdict,metric-sample}.schema.jsonheld to field, required-set, bound, and enum-string parity by the contract tests. A spec's decision bounds are intersected with the policy envelope, so a spec can tighten its own gate but never loosen it.crates/control-planeadditively with an append-onlyexperiment_trialstable andsnapshot/record_trial/read_trial/trial_ids, plus anUnknownTrialerror; the existing journal schema and broker lifecycle are unchanged. Integration and acceptance-transcript tests cover a promoted trial, a rejected trial that leaves the baseline untouched, journal-only replay reproducing the identical verdict, tampered-row rejection, and the MVP acceptance criterion. README,docs/ARCHITECTURE.md,docs/IMPLEMENTATION_PLAN.md, ADR 0002, the threat model, andAGENTS.mddocument the engine, the replay gate and exactly what it does and does not detect, and the safe-alpha limitation that the lifecycle always rolls back so the verdict gates whether a candidate is applied rather than whether it persists.Risk Assessment
✅ Low: The change is large but strictly additive, unprivileged, and mock-only - it touches no broker write path, no privileged code, and no existing schema semantics beyond formalizing the 300s lease cap the broker already enforced - and after six review rounds the run-time and replay gates are mutually consistent with no correctness defect found, leaving only two documentation-accuracy items.
Testing
I ran the whole workspace test suite (all green), then drove the feature the way an operator would: the experiment-runner demo CLI twice in separate processes to confirm bit-identical deterministic output, and a new transcript-producing acceptance test over a single durable on-disk SQLite journal that shows a measured experiment promoted through the full snapshot/preview/apply/verify/rollback lifecycle (+30.0 fps, verified and rolled back, provider returned to mock.value = 10), a second measured experiment rejected on the 80 C ceiling despite a larger +60.0 fps gain with the baseline provably untouched, the persisted experiment_trials rows read with a plain SQLite connection including trial 1's verbatim 1499-byte self-describing payload, a replay from a brand-new ControlPlane handle with the provider parked at an unrelated value that recomputes both verdicts identically and reports them policy-legal, and a widened-bounds copy that replays with the same verdict but is flagged policy-illegal naming max_temperature_c - proving the replay gate is a real check. I also confirmed the contracts crate enforces field, bound, and enum-string parity between the typed spec/verdict and the checked-in JSON schemas. One environmental blocker had to be worked around first: this machine's global cargo config pins CC to zig-cc, which cannot compile libsqlite3-sys, so I used temporary cc/ar shims inside the gitignored target/ dir and deleted them afterwards. Per the testing rules I did not run fmt, clippy, or CI's check-jsonschema step, so the test file I added has not been through the formatter. This change has no UI, HTML, or rendered surface - the end-user surface is a CLI and a SQLite journal, so the evidence is a CLI transcript plus persisted database state rather than screenshots. Overall result: everything passes and the stated user intent is demonstrated end-to-end.
Evidence: Acceptance transcript: promote, reject, persisted journal rows, journal-only replay, tampered-row gate
provider parked at mock.value = 10 [promoted] trial id 1 hypothesis : raising mock.value from 10 to 40 improves FPS within thermal and power limits baseline value : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples candidate value : 40 -> mean 100.0 fps, 70.0 C, 140.0 W, 0 errors over 5 samples bounds : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples verdict : Promote (Promoted), fps_improvement = +30.0 lifecycle : provider mock, preview "set mock.value from 10 to 40", verified = true, rolled_back = true provider after : mock.value = 10 (the lease restored the pre-state) [rejected] trial id 2 hypothesis : raising mock.value from 10 to 70 improves FPS within thermal and power limits baseline value : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples candidate value : 70 -> mean 130.0 fps, 85.0 C, 170.0 W, 0 errors over 5 samples bounds : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples verdict : Reject (TemperatureExceeded), fps_improvement = +60.0 lifecycle : none - the candidate was never applied provider after : mock.value = 10 (untouched - nothing was applied) persisted experiment_trials rows: id 1, recorded_at 2026-07-27T10:45:45.919Z, payload 1499 bytes id 2, recorded_at 2026-07-27T10:45:45.920Z, payload 1416 bytes replay from a fresh handle (provider now at mock.value = 0, no spec in hand, no chat history): [promoted] trial 1: recorded Promote (Promoted) -> recomputed Promote (Promoted); identical = true, policy legal = true [rejected] trial 2: recorded Reject (TemperatureExceeded) -> recomputed Reject (TemperatureExceeded); identical = true, policy legal = true [tampered] trial 3: bounds widened to 200 C; recomputed verdict still identical = true, but policy legal = false (experiment spec rejected: max_temperature_c is 200, outside the 0 exclusive to 90 inclusive the policy allows) acceptance criterion: 2 measured experiments reached a decision through the immutable evaluator - Promote and Reject - and both re-evaluate identically from the journal aloneEvidence: Verbatim persisted TrialRecord payload of the promoted trial (read from SQLite)
Evidence: Demo CLI run twice in separate processes - identical output proves determinism
$ cargo run -p fpsmaxxing-experiment-runner # run 1 trial 1 -> Promote (Promoted); fps_improvement = 30.0 lifecycle: provider mock, verified = true, rolled_back = true replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true exit=0 $ cargo run -p fpsmaxxing-experiment-runner # run 2 (separate process, no shared state) trial 1 -> Promote (Promoted); fps_improvement = 30.0 lifecycle: provider mock, verified = true, rolled_back = true replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true exit=0Evidence: Named behaviour coverage: schema parity, evaluator known-answer fixtures, integration scenarios
Evidence: Full workspace test run summary
Pipeline
Updates from git push no-mistakes
... (3 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 Fix: gate lease, lifecycle result fields, and record version
6 issues (1 warning, 5 infos) still open:
crates/control-plane/src/lib.rs:297- The broker's lease ceiling is now enforced by an untested branch.a_promoted_trial_survives_a_lifecycle_the_broker_refuses(integration.rs:223) was the only test in the workspace that reachedPolicyDenied("lease exceeds 300 seconds"), and this commit correctly re-levered it to anApprovalRequiredrisk class because the runner rejects an oversized lease before measurement now. Nothing replaced the lost coverage:crates/control-plane's own test module never had a lease case (verified against base 7f3ab1d), and the gateway e2e only covers the value bound (mcp_e2e.rs:164, value 101). The branch is still reachable - the gateway'sfpsmaxxing.run_mock_lifecycletool acceptslease_secondsstraight from the LLM and hands it torun_lifecycle- so an off-by-one (>vs>=) or a change toMAX_LEASE_SECONDSwould ship silently. The TTL lease is what bounds how long a mutation may persist, one of the repo's non-negotiable boundaries. Fix is mechanical: a unit test in the control-plane test module assertingrun_lifecycledeniesMAX_LEASE_SECONDS + 1and acceptsMAX_LEASE_SECONDS.apps/gateway/src/lib.rs:67-MAX_LEASE_SECONDSwas introduced with the stated purpose that callers hold the lease "to the same ceiling this policy does rather than restating it", andMAX_MOCK_VALUEsays the same. The gateway'stools/listresponse still restates both as literals:"value": {"maximum": 100}and"lease_seconds": {"maximum": 300}. That copy is the one the LLM reads to decide what to request, and no test pins it to the constants, so tightening policy would leave the advertised schema over-promising a value the broker then denies.apps/gatewayalready depends onfpsmaxxing-control-plane, so this is a mechanical substitution into thejson!macro. (sidecars/mock-provider/src/lib.rs:43also hardcodesmaximum: 100, but a sidecar importing the control plane would invert the layering - leave that one.)schemas/experiment.schema.json:31-validate_specnow rejectslease_seconds > 300, but the published spec schema declares{"type": "integer", "minimum": 1}with no maximum, so an LLM authoring a spec against the contract can produce a schema-valid document the runner refuses withInvalidSpec. Every other policy bound on a typed spec field is mirrored there and pinned by a contract test -MAX_SAMPLESon the three sample counts, and eachDecisionBoundsthreshold viadecision_bounds_are_bounded_like_the_schema.MAX_MOCK_VALUEis unmirrored only because it lives inside the opaqueparametersobject;lease_secondsis a first-class typed field, so the same excuse does not apply. Mirroring it properly means deciding where the constant belongs: a#[schemars(range(max = ...))]onChangeRequest.lease_secondswould moveMAX_LEASE_SECONDSintocrates/contractsand apply it to everyChangeRequest, not just experiment targets - which is a real design call, hence flagging rather than fixing.docs/ARCHITECTURE.md:28- Three places enumerate exactly what the replay policy gate re-checks, and none were updated when the lease bound joined it in the final commit: ARCHITECTURE.md:28 ("a row whose spec, capability, declared sample counts, candidate value, or baseline was rewritten"), ADR 0002 lines 27 and 30 ("capability, sample counts, decision bounds, and candidate value" / "the spec, the capability, the declared sample counts, the candidate value, the baseline ceiling, and the lifecycle fields"), and themain.rs:10-11module doc that lists the same fields for the demo's exit-code contract. ARCHITECTURE.md:27 likewise lists what is bounded before measurement without the lease. These enumerations are the auditable description of the tamper gate, anda_replay_reports_a_record_the_policy_gate_would_refusenow has a lease case the docs do not describe.apps/experiment-runner/src/runner.rs:429- Noting the coupling rather than asking for a change: the new gate requiresrolled_back == trueon every recorded lifecycle, which is sound today becauserun_stageshard-codesrolled_back: trueon its onlyOkpath (control-plane/src/lib.rs:347). Butrolled_backmeans "the captured baseline was restored", and the documented next step is durable keep-or-rollback under the privileged broker - at which point a legitimately kept promotion carriesrolled_back: falseand this audit gate flags it as tampered with. Archived rows stay legal (they all carrytrue), so the migration is a gate relaxation rather than an audit break, but it is a forward-compat coupling worth knowing about when the keep path lands. Same forverified, though that one is a genuine invariant ofobserve_appliedrather than an artifact.apps/experiment-runner/src/runner.rs:287- On the lifecycle-failure pathrun_trialjournals the record, then returnsErr(error)and dropsjournaled- the id of the row it just wrote. The doc promises the record is journaled before the error surfaces, but a caller cannot address it: recovering the id means scanningtrial_ids()and assuming the last entry is theirs, which is what integration.rs:243 does viaids[0]. That assumption holds only single-threaded; two processes sharing one journal file (a supported configuration -begin_experimenttakes an immediate transaction with a busy timeout specifically to make concurrent gateways safe) can interleave appends, and the recovering caller has no way to identify its own row. Carrying the id on the error variant, or returningResult<StoredTrial, (i64, RunnerError)>, would close it - a public signature change, so flagging rather than fixing.🔧 Fix: test lease gate, share lease cap, carry trial id
5 infos still open:
apps/experiment-runner/src/runner.rs:118- ATrialRecordcarries no link to the lifecycle journal rows it authorized.run_lifecycleallocates a correlation id inbegin_experiment(control-plane/src/lib.rs:396) and writes eight stage rows under it, butLifecycleResult(control-plane/src/lib.rs:97-105) does not return that id, soLifecycleOutcomecannot record it and no public API exposes it (journal_stages()returns stage names only). An auditor holding a trial row therefore cannot find theexperiment_journalrows for the mutation it authorized, or vice versa, except by correlatingrecorded_attimestamps across the two tables - which is ambiguous in exactly the configuration the design explicitly supports: concurrent gateways sharing one journal file (ADR 0002 consequences). The crash story still works (a danglingapply-intentwith no trial row is whatdoctorreports), so this is forward correlation only. Closing it means adding anexperiment_idtoLifecycleResultand mirroring it onLifecycleOutcome- a public control-plane type change, which the task deliberately kept additive and minimal, hence flagging rather than fixing.apps/experiment-runner/src/runner.rs:304- This commit restructured the lifecycle-failure path specifically to stop callers guessing at journal state, but the secondary failure on that path is still unstructured: whenrecord_trialfails, theControlPlaneErroriseprintln!'d from a library crate and discarded, and the caller receives onlytrial_id: None. That is the more serious condition of the two - the measurements that authorized a mutation were lost entirely, not merely accompanied by a broker refusal - yet it is the only one a programmatic consumer cannot inspect: a disk-full journal, a serialization fault, and a locked database are indistinguishable. The precedence rule (lifecycle error wins) is right and documented; carrying the journal error alongside it, rather than only to stderr, would finish what thetrial_idchange started. Public error-type change, so flagging rather than fixing.apps/experiment-runner/src/runner.rs:406-check_policyre-runs the currentvalidate_specover a journaled spec, so tightening any policy constant retroactively marks every archived row recorded under the looser ceiling aspolicy_legal = false- reported to an auditor as "the journaled row is not one this runner would have written" and exiting the demo non-zero (main.rs:76). LoweringMAX_DECISION_TEMPERATURE_Cfrom 90 to 85 would flag every historical trial withmax_temperature_c: 88; the same holds forMAX_LEASE_SECONDS,MAX_MOCK_VALUE, andMAX_SAMPLES, and the lease joining the gate in this branch widens the surface.TRIAL_RECORD_VERSIONdoes not cover it: a constant change is not a record-format change, so nothing forces a bump. This is the same false-alarm class the authors already reasoned about and rejected for the provider manifest (runner.rs:375-380, ADR 0002 final rationale line), left unaddressed for the constants themselves. Options are to journal the envelope with the record, or to document the constant-drift case next to the manifest paragraph in ADR 0002 and treat a constant change as a version bump - a real design call.crates/contracts/src/lib.rs:205-ExperimentSpec.hypothesisis an unboundedStringwritten verbatim into a single durable journal row byrecord_trial. Every other LLM-authored field on the spec carries a ceiling declared in this crate and mirrored inschemas/experiment.schema.json- the three sample counts againstMAX_SAMPLES, eachDecisionBoundsthreshold, and nowlease_seconds- and the schema giveshypothesisaminLength: 1but nomaximum. There is no LLM-reachable path torun_trialtoday (the gateway exposes onlyrun_mock_lifecycle), so this is a latent rather than live concern, but the spec is designed to be agent-authored and the row size is whatever the author sends. AmaxLengthin the schema plus a matching check invalidate_specwould follow the pattern the sample counts already set; choosing the limit is a product decision.crates/contracts/src/lib.rs:71- The newMAX_LEASE_SECONDSdoc enumerates where the ceiling is mirrored -schemas/experiment.schema.json, the broker policy, the experiment runner - and omits the fourth site:apps/gateway/src/lib.rs:67restates"lease_seconds": { "maximum": 300 }as a literal in thetools/listschema an LLM reads to decide what to request. The same commit added exactly that caveat toMAX_MOCK_VALUE("Callers that publish their own request schema, such as the gateway's advertised tool input, state the bound independently", control-plane/src/lib.rs:19-20), so the asymmetry leaves the lease's known drift risk with no in-code pointer: tightening the constant would leave the advertised schema over-promising a value the broker then denies, and no test pins the two together. One sentence mirroring theMAX_MOCK_VALUEcaveat; no gateway edit needed.🔧 Fix: carry journal error, bound hypothesis, document policy drift
4 issues (1 warning, 3 infos) still open:
docs/adr/0002-alpha-experiment-journal.md:35- The new policy-drift paragraph prescribes a remedy that destroys the signal it exists to preserve. Lines 33-34 establish thatpolicy_legalreports a row against the policy in force now, so tighteningMAX_LEASE_SECONDS,MAX_MOCK_VALUE,MAX_SAMPLES,MAX_HYPOTHESIS_CHARS, or a decision-bound ceiling deliberately flags every archived row recorded under the looser one - "an auditor wants them surfaced rather than grandfathered". Line 35 then makes a policy-constant change aTRIAL_RECORD_VERSIONbump "so a flagged archive is attributable to the constant that moved". But the version gate is fail-closed and runs first:replay_trial(runner.rs:355-362) readsschema_versionoff the raw payload and returnsUnsupportedRecordVersionbeforecheck_policyis ever called, so after the bump every pre-existing row errors out of replay instead of replaying withpolicy_legal = false. Nothing is flagged - the whole archive becomes unreadable, andmain.rs:69propagates that as a hardErrrather than the non-zero policy-gate exit at main.rs:84. The same claim is repeated in thecheck_policydoc (runner.rs:390-391). The two mechanisms are also in tension by design: the version is documented at ADR line 26 as tracking "what a row contains" (a field addition a reader must refuse), while line 35 explicitly says a constant change is "a change in what the journal means rather than in what a row contains" - and then routes it through the refusal mechanism anyway. Either the constant drift needs its own attribution (journaling the envelope with the record, or a separate policy-generation field the gate reads without refusing the row), or the ADR should state plainly that a constant bump retires the existing archive from replay.apps/experiment-runner/src/runner.rs:384- Four enumerations now list the hypothesis among the fields whose rewrite replay detects:check_policy's doc ("a row whose capability, hypothesis, sample counts, decision bounds, candidate value, or TTL lease were rewritten is reported as illegal"), ARCHITECTURE.md:28, ADR 0002 lines 27 and 30, and themain.rs:11module doc that defines the demo's exit-code contract. Only the length ceiling is actually checked (runner.rs:557-562). Every other field in those lists has a second term to check against - the capability has exactly one legal value,candidate_valueand the sample counts are cross-checked against redundant copies in the record, the lease and bounds have policy ceilings that a rewrite must clear - so an arbitrary rewrite of any of them trips the gate. The hypothesis has none: it is free text with no redundant copy, so rewriting "raise mock.value to 40" to "lower mock.value to 40" replays aspolicy_legal = true. That is the field a tamperer would most want to rewrite, because it is the human-readable statement of what a promotion was for, and the docs now assert it is protected when only its length is. The fix is wording: say the hypothesis is bounded rather than that its rewrite is detected, and keep the ADR:30 "detection is structural only" caveat honest about which structure exists.apps/experiment-runner/src/runner.rs:557-schemas/experiment.schema.json:16declares"hypothesis": { "type": "string", "minLength": 1, "maxLength": 4096 }, but only the ceiling is enforced in Rust.validate_speccheckschars().count() > MAX_HYPOTHESIS_CHARSand nothing else, and#[schemars(length(max = MAX_HYPOTHESIS_CHARS))](contracts/src/lib.rs:217) emits nominLength, so the generated schema and the checked-in one disagree on the floor - the parity test only compares property names,required, andadditionalProperties, so it does not catch the difference, andthe_hypothesis_is_bounded_like_the_schemapins onlymaxLength. Consequences: a spec withhypothesis: ""is accepted, measured, and journaled despite the published contract forbidding it, and on replay a row whose hypothesis was blanked out passes the gate that now refuses one that was lengthened. Blanking is the cheaper tamper of the two. Addinglength(min = 1)plus a matching check would close both halves; it is a small behavior change on a wire contract, so it is worth confirming rather than applying silently.apps/experiment-runner/src/runner.rs:326- This commit made the lifecycle-failure path fully inspectable -LifecycleFailednow carries the trial id or the reason the record was lost - but the mirror case is untouched. When a promotion's lifecycle succeeds andrecord_trialthen fails,journaled?returns a bareRunnerError::ControlPlane(ControlPlaneError::Journal(_))with no indication that a lifecycle ran at all. That is the case where the provider was actually mutated (apply, verify, rollback all completed) and the measurements authorizing it were lost, yet the caller learns strictly less than in the refused-lifecycle case where nothing reached the provider.doctorcannot recover it either: it flags experiments with anapply-intentand no terminal record (ADR 0002 consequences), and this experiment has acompletedrecord, so a completed lifecycle with no trial row is invisible to both readers. Today the state impact is nil becauserun_lifecyclealways restores the pre-state, but the documented next step is durable keep-or-rollback under the privileged broker - at which point this is a knob left mutated with no trial record and no dangling-intent flag. Carrying the lifecycle outcome on the error, the way the failure path now carries the journal error, would close it; that is another public error-type change, hence flagging rather than fixing.🔧 Fix: fix policy-drift docs, hypothesis floor, and lost-record lifecycle
4 infos still open:
apps/experiment-runner/src/runner.rs:639- With the hypothesis now bounded at both ends,spec.target.parametersis the field that inherits the problem the hypothesis bound was added to solve.validate_specreads exactly one key out of it (candidate_value->parameters["value"], runner.rs:726-732) and bounds that againstMAX_MOCK_VALUE; every other key passes through unexamined.parametersis a bareserde_json::Value(contracts/src/lib.rs:86) whose schema is{"type": "object"}with nomaxProperties,maxLength, or depth limit, so{"value": 40, "pad": "<10 MB>"}is accepted, measured, and written verbatim - and written twice durably, once into the trial row byrecord_trialand again into the lifecycle journal'sapply-intentrecord byrun_stages(control-plane/src/lib.rs:324), sincerun_lifecycleserializes the wholeChangeRequest.validate_spec's own doc (runner.rs:572-573) states "Everything the measurement phase and the decision gate consume arrives over the wire, so all of it is bounded before any measurement work runs", which is now true of every spec field except this one. Reachability matches the hypothesis case exactly: latent, not live - the gateway buildsparametersitself from a validated integer (gateway/src/lib.rs:90-91), so there is no LLM-reachable path to an arbitrary object today, butExperimentSpecis designed to be agent-authored. Closing it means bounding a field on the sharedChangeRequest(a size cap, or requiringparametershold only the keys the capability'sinput_schemadeclares), which affects the broker path too - a design call rather than a mechanical fix.apps/experiment-runner/src/runner.rs:607- The floor isspec.hypothesis.chars().count() < 1, so it catches exactly the empty string. Its stated purpose is that "a promotion cannot be journaled with no statement of what it was for" (runner.rs:590-591) and, on the replay side, that "blanking the statement of what a promotion was for is the cheaper rewrite of the two" (integration.rs:683-686). A single space defeats both:hypothesis: " "is one character, passesvalidate_spec, passescheck_policy, and carries exactly as much meaning as"". This matters more on the replay side than the run-time side - the run-time caller has no motive to send a space, but a tamperer rewriting a journaled row to erase what a promotion was for now simply writes" "instead of""and the gate that was just added to catch that reportspolicy_legal = true. The checked-in schema has the same hole (minLength: 1admits" "), so keeping the two aligned means either adding a"pattern": "\\S"alongside atrim().is_empty()check, or accepting the limit and saying in the doc that the floor bounds length rather than substance - the same distinction this commit just drew for the ceiling.apps/experiment-runner/src/runner.rs:449- Three places now justify not detecting a sample rewrite by saying it would need an external anchor:check_policy's doc ("catching that needs each row anchored outside itself - a signed or hash-chained journal"), ARCHITECTURE.md:29, and ADR 0002 line 32. That reason does not hold for the alpha as built.model::measureis a pure function of(value, warmup, counted)(model.rs:52-62) and every one of those three inputs is already in the record:baseline_value/candidate_valueandspec.warmup_samplesplus the two declared counts. Sorecord.baseline_samples == model::measure(record.baseline_value, record.spec.warmup_samples, record.spec.baseline_samples.get())is an invariant of every row this runner wrote, checkable with no anchor at all - the same shape as the count checkholds_declared_countalready performs one field over. A coherent rewrite of the measurements plus the verdict, which the docs name as the one surviving tamper, would be caught outright. Whether to add it is a genuine call: replay already consultsMODELED_CAPABILITY_ID, but re-deriving contents couples the audit gate to the measurement model in a way that must be deleted the moment real PresentMon telemetry lands (real samples are not reproducible, which is the whole reason they are journaled verbatim). Either add the check while the model is pure, or correct the stated reason from "needs a hash chain" to "deliberately not coupled to the measurement model, so the alpha does not exploit its reproducibility".apps/experiment-runner/src/main.rs:55-TrialNotJournaledwas added specifically so a caller learns that a promotion reached the provider and its record was lost, but the demo - the one caller in the tree - falls through toErr(error) => return Err(error), whichTerminationrenders as a derivedDebugdump rather than the crafted Display ("trial record was lost after its promotion completed the lifecycle"). The siblingLifecycleFailedgets nine lines of tailored reporting immediately above (main.rs:36-54). The module doc at main.rs:14-17 still enumerates three failure modes and says the demo reports how a lost row was lost, which is now the case for one of the two lost-record paths. Exit status is unaffected - returningErrfrommainis non-zero either way - and the demo's:memory:journal makes the path unreachable in practice, so this is about the enumeration matching the code rather than a behavioral gap. A fourth match arm printingsourceand whether a lifecycle ran, plus a sentence in the module doc, closes it.🔧 Fix: bound target parameters, correct sample-integrity docs, report lost record
2 issues (1 warning, 1 info) still open:
apps/experiment-runner/src/runner.rs:451- The replay gate's four enumerations claim more detection thancheck_policyperforms, and the claim is load-bearing because it is the auditable description of what a clean replay proves.check_policycross-checksrecord.candidate_valueagainst the value it re-derives fromrecord.spec, and boundsrecord.baseline_valuebyMAX_MOCK_VALUE. Neither is tied to the recorded samples. Two consequences:(1) Rewriting
spec.target.parameters.valueandcandidate_valuetogether to any other in-bounds value is undetected. Set both to 100 on a trial actually measured at 40:validate_specreturns 100,100 == record.candidate_value, the samples and verdict are untouched sois_consistent()holds, and the row replays withpolicy_legal = true. The archive then states that value 100 was measured at 70 C and promoted, when those are the value-40 samples. This needs no sample forgery at all - two field edits.(2)
baseline_valuehas no second term anywhere; only the ceiling. Rewriting it from 10 to 90 passes replay outright, yet ARCHITECTURE.md:28 lists "or baseline was rewritten after the fact is reported as outside policy". (main.rs:11and the ADR correctly say "baseline ceiling" - it is ARCHITECTURE.md that over-claims.)The sentence to fix is runner.rs:451-452: "every other checked field has a second term to disagree with, while the hypothesis is free text with no redundant copy in the record." That is false for
baseline_value, and forcandidate_valuethe second term is a copy of the same claim rather than an independent one - so a coherent edit of the pair survives. The caveat paragraph names the surviving tamper as "a rewrite of the measurements together with the verdict they imply, or of the hypothesis text within its bounds", which understates it: relabelling the knob value requires neither.This is the same class as the hypothesis wording corrected in round 4, and it is exactly what the sample re-derivation check would have caught - the round-5 decision not to re-derive is sound (real telemetry is not reproducible), but the docs should stop describing the resulting gap as covered. Note
lifecycle.previewdoes embed the applied value for a promoted row and is unchecked, but it is provider-authored free text, so parsing it is not a serious cross-check.Suggested fix is wording only, in all four places: say the candidate value is cross-checked against its own spec and the baseline is bounded, and extend the surviving-tamper caveat to name a coherent relabel of the candidate value and a rewrite of the baseline within its ceiling.
apps/experiment-runner/src/model.rs:29-MODELED_PARAMETERS = ["value"]is now the runner's authority on which target parameters are acceptable, but no contract an agent can read says so, and the same fact is already declared elsewhere.sidecars/mock-provider/src/lib.rs:41-45advertisesinput_schema: {"type": "object", "required": ["value"], "properties": {"value": {"type": "integer", "minimum": 0, "maximum": 100}}}- the capability's own published statement of what it takes - and it omitsadditionalProperties: false.schemas/experiment.schema.json:30declaresparametersas a bare{"type": "object"}. So both published surfaces permit extra keys whilevalidate_specrefuses them withInvalidSpec. An agent authoring a spec against either contract can produce a schema-valid document the runner rejects, which is the same divergence the round-2lease_secondsfinding closed for a typed field.Nothing validates against
input_schemaat runtime (verified: it is only read bymcp_e2e.rs:84for an assertion), so this is a contract-consistency issue rather than a live bug, and there is no LLM-reachable path torun_trialtoday.The narrow fix that respects the round-5 instruction not to touch the shared
ChangeRequestschema or the broker path is to add"additionalProperties": falseto the mock capability'sinput_schema, keeping the declaration with the capability that owns it. That leavesMODELED_PARAMETERSas a deliberate second copy - justified, since replay must not consult the attached provider's manifest (a_replay_does_not_depend_on_the_attached_provider) - but at least the published contract would agree with the runner. Flagging rather than fixing because it tightens a published sidecar contract.🔧 Fix: correct replay-gate docs on candidate and baseline detection
2 infos still open:
apps/experiment-runner/src/runner.rs:448- The enumerations rewritten in the final commit claim exhaustiveness they do not have.check_policyinspects exactly two of the recorded lifecycle's four fields (verified,rolled_back, runner.rs:528-535) and none oflifecycle_error's two. Sorunner.rs:448- "A recorded lifecycle is held to its own contents too" - over-claims, and the "three rewrites" list at runner.rs:456-464, ARCHITECTURE.md:30, ADR 0002:31, and main.rs:15-19 misses a fourth class.Concretely: on a promoted row, setting
lifecycle.provider_idto any other string, or rewritinglifecycle.preview, decodes cleanly (LifecycleOutcometypes them as free-textString), leaves the presence andverified && rolled_backchecks satisfied, leaves the verdict untouched, and replays aspolicy_legal = true. Same for a refused promotion:lifecycle_error.kindcan be moved from "policy-denied" to "provider" andlifecycle_error.errorrewritten wholesale, with nothing to contradict it.This matters on the terms the ADR itself sets at line 29 - "the trial row is the only auditable statement that a promotion reached the provider". Which provider owned the mutation, and why the broker refused one, are part of that statement, and both are rewritable. Like the candidate-value gap corrected in the same commit, it cannot be closed by a second term:
provider_idcould only be cross-checked against the attached manifest, whicha_replay_does_not_depend_on_the_attached_providerdeliberately forbids.Fix is wording in the same four places: say the recorded lifecycle is held to
verifiedandrolled_backrather than to "its own contents", and add the lifecycle's provider and preview text plus the whole recorded broker error to the list of rewrites that survive.docs/README.md:19- CLAUDE.md directs every agent session to start atdocs/README.md, and its one-line summary for ADR 0002 still reads "Write-ahead apply intent, terminal outcomes, and deferred two-phase journaling". This branch grew that ADR by ~20 lines covering theexperiment_trialstable, the append-only trial-record contract, the replay tamper gate and exactly what it detects, and the policy-constant drift rule - and it explicitly supersedes one of the deferrals the blurb advertises ("This supersedes the original deferral of a dedicated experiments table", ADR line 13). A reader scanning the index for where the trial-journal and replay-gate decisions are recorded would not find them. One clause added to the blurb.✅ **Test** - passed
✅ No issues found.
cargo test --workspace(all suites green: 15 contracts, 14 control-plane, 29 experiment-runner unit, 21 experiment-runner integration, 2 gateway mcp_e2e, 1 mock-provider)cargo run -p fpsmaxxing-experiment-runnerrun twice in separate processes - byte-identical output, confirming the evaluator/model are deterministic and wall-clock-freecargo test -p fpsmaxxing-experiment-runner --test acceptance_transcript -- --nocapture(new transcript test I added: promote -> reject -> persisted-journal dump -> journal-only replay -> tampered-row gate)cargo test -p fpsmaxxing-experiment-runner --test integration(named coverage incl.mvp_one_measured_experiment_is_promoted_or_rejected_by_the_evaluator,a_promoted_experiment_runs_the_full_lifecycle,a_rejected_experiment_is_never_applied_and_leaves_the_baseline,a_trial_replays_from_the_journal_alone_with_an_identical_verdict)cargo test -p fpsmaxxing-experiment-runner --lib(evaluator known-answer fixtures for each ordered threshold, measurement-model determinism, spec/policy-envelope gate)cargo test -p fpsmaxxing-contracts(typed ExperimentSpec/Verdict/MetricSample vs schemas/*.json field, required-set, bound, and enum wire-string parity)Manual: read the persistedexperiment_trialsrows with a plainrusqlite::Connectionon the journal file to show durable state rather than describe itEnvironment fix: built with throwawaycc/arzig shims in the gitignoredtarget/dir (global cargo config pins CC=zig-cc, which rejects cc-rs's 4-component triple), then removed themdocs/IMPLEMENTATION_PLAN.md:126- The plan's MVP acceptance criteria and sprint phase exit gates have no convention for marking a criterion as satisfied, so the now-demonstrated criterion 'One measured experiment is promoted or rejected using the immutable evaluator' (and the 'Experiment engine' phase gate 'Trial replays without chat history') still read as pending. Introducing a status marker convention is an editorial decision for a human; I documented the implemented behavior in the Experiment loop section instead.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.