Skip to content

feat(experiment-runner): add deterministic trial engine with immutable evaluator and journal replay - #11

Open
undeemed wants to merge 16 commits into
mainfrom
fm/fpsm-exp-jd
Open

feat(experiment-runner): add deterministic trial engine with immutable evaluator and journal replay#11
undeemed wants to merge 16 commits into
mainfrom
fm/fpsm-exp-jd

Conversation

@undeemed

Copy link
Copy Markdown
Owner

Intent

Implement the experiment-engine phase of fpsmaxxing: baseline/candidate trials plus a decision gate, per docs/IMPLEMENTATION_PLAN.md. Deliverables: (1) a typed experiment spec (hypothesis text, target capability ChangeRequest, warmup, repeated baseline/candidate measurement counts, and correctness/constraint bounds), serde+schemars-typed in crates/contracts with schemas/{experiment,verdict}.schema.json kept in sync (contract tests enforce field/enum parity); (2) a deterministic, immutable evaluator - a pure function of recorded baseline+candidate samples and fixed DecisionBounds, no LLM and no wall-clock, applying an ordered fixed-threshold rule (minimum sample count, then temperature/power/error ceilings, then mean-FPS improvement threshold), unit-tested with known-answer fixtures; (3) a trial runner in apps/experiment-runner that measures a baseline, measures the candidate, and runs the candidate through ControlPlane::run_lifecycle gated by the evaluator verdict, journaling a self-describing TrialRecord so the trial replays and re-evaluates from the journal alone without chat history; (4) integration tests against sidecars/mock-provider for a promoted trial, a rejected trial that leaves the baseline untouched, and a journal-only replay reproducing the identical verdict; (5) the MVP acceptance criterion 'one measured experiment is promoted or rejected using the immutable evaluator', demonstrated in a test.

Key decisions a diff-only reviewer would miss: Control-plane changes were deliberately kept additive and minimal per the task constraint - a new experiment_trials table plus read/journal APIs (snapshot, record_trial, read_trial, trial_ids); the existing journal schema and broker lifecycle were NOT restructured. Safe-alpha limitation (intentional and documented in runner.rs and the commit body): mock capabilities are leased and ControlPlane::run_lifecycle always restores the pre-state before returning, so the verdict gates whether the candidate is applied at all (Promote exercises the full snapshot/preview/apply/verify/rollback lifecycle; Reject never mutates the knob) rather than driving physical persistence; durable keep-or-rollback awaits the privileged broker. The measurement model is a deterministic stand-in for PresentMon/hardware telemetry, which is unavailable on this Linux aarch64 safe-alpha path - it is a pure function of the knob value so trials replay bit-identically. No heavyweight statistics dependencies were added (deny.toml gates deps); the decision rule is intentionally a simple documented fixed-threshold comparison of means with a minimum sample count. Linux-only, mock provider is the provider. A sibling task (fpsm-wdog-vv) is concurrently adding watchdog journal-read logic on a separate branch; my control-plane additions are purely additive to avoid conflict.

What Changed

  • Added apps/experiment-runner as a library plus demo binary: a deterministic measurement model standing in for live telemetry, a pure evaluate function applying an ordered fixed-threshold rule (minimum sample count, then temperature/power/error ceilings, then mean-FPS improvement) with no clock or LLM input, and a trial runner that measures a baseline and a candidate, runs ControlPlane::run_lifecycle only on a Promote verdict, and appends a versioned self-describing TrialRecord carrying the spec, samples, verdict, and the lifecycle outcome or broker error. replay_trial re-evaluates a record from the journal alone and re-applies the full policy gate, reporting whether the row is one this runner could have written.
  • Added the experiment wire contracts in crates/contracts (ExperimentSpec, DecisionBounds, MetricSample, MetricSummary, Decision, VerdictReason, Verdict) with policy-owned bound constants (MAX_LEASE_SECONDS, MAX_SAMPLES, MIN/MAX_HYPOTHESIS_CHARS, and the decision ceilings), and new schemas/{experiment,verdict,metric-sample}.schema.json held to field, required-set, bound, and enum-string parity by the contract tests. A spec's decision bounds are intersected with the policy envelope, so a spec can tighten its own gate but never loosen it.
  • Extended crates/control-plane additively with an append-only experiment_trials table and snapshot/record_trial/read_trial/trial_ids, plus an UnknownTrial error; the existing journal schema and broker lifecycle are unchanged. Integration and acceptance-transcript tests cover a promoted trial, a rejected trial that leaves the baseline untouched, journal-only replay reproducing the identical verdict, tampered-row rejection, and the MVP acceptance criterion. README, docs/ARCHITECTURE.md, docs/IMPLEMENTATION_PLAN.md, ADR 0002, the threat model, and AGENTS.md document the engine, the replay gate and exactly what it does and does not detect, and the safe-alpha limitation that the lifecycle always rolls back so the verdict gates whether a candidate is applied rather than whether it persists.

Risk Assessment

✅ Low: The change is large but strictly additive, unprivileged, and mock-only - it touches no broker write path, no privileged code, and no existing schema semantics beyond formalizing the 300s lease cap the broker already enforced - and after six review rounds the run-time and replay gates are mutually consistent with no correctness defect found, leaving only two documentation-accuracy items.

Testing

I ran the whole workspace test suite (all green), then drove the feature the way an operator would: the experiment-runner demo CLI twice in separate processes to confirm bit-identical deterministic output, and a new transcript-producing acceptance test over a single durable on-disk SQLite journal that shows a measured experiment promoted through the full snapshot/preview/apply/verify/rollback lifecycle (+30.0 fps, verified and rolled back, provider returned to mock.value = 10), a second measured experiment rejected on the 80 C ceiling despite a larger +60.0 fps gain with the baseline provably untouched, the persisted experiment_trials rows read with a plain SQLite connection including trial 1's verbatim 1499-byte self-describing payload, a replay from a brand-new ControlPlane handle with the provider parked at an unrelated value that recomputes both verdicts identically and reports them policy-legal, and a widened-bounds copy that replays with the same verdict but is flagged policy-illegal naming max_temperature_c - proving the replay gate is a real check. I also confirmed the contracts crate enforces field, bound, and enum-string parity between the typed spec/verdict and the checked-in JSON schemas. One environmental blocker had to be worked around first: this machine's global cargo config pins CC to zig-cc, which cannot compile libsqlite3-sys, so I used temporary cc/ar shims inside the gitignored target/ dir and deleted them afterwards. Per the testing rules I did not run fmt, clippy, or CI's check-jsonschema step, so the test file I added has not been through the formatter. This change has no UI, HTML, or rendered surface - the end-user surface is a CLI and a SQLite journal, so the evidence is a CLI transcript plus persisted database state rather than screenshots. Overall result: everything passes and the stated user intent is demonstrated end-to-end.

Evidence: Acceptance transcript: promote, reject, persisted journal rows, journal-only replay, tampered-row gate

provider parked at mock.value = 10 [promoted] trial id 1 hypothesis : raising mock.value from 10 to 40 improves FPS within thermal and power limits baseline value : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples candidate value : 40 -> mean 100.0 fps, 70.0 C, 140.0 W, 0 errors over 5 samples bounds : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples verdict : Promote (Promoted), fps_improvement = +30.0 lifecycle : provider mock, preview "set mock.value from 10 to 40", verified = true, rolled_back = true provider after : mock.value = 10 (the lease restored the pre-state) [rejected] trial id 2 hypothesis : raising mock.value from 10 to 70 improves FPS within thermal and power limits baseline value : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples candidate value : 70 -> mean 130.0 fps, 85.0 C, 170.0 W, 0 errors over 5 samples bounds : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples verdict : Reject (TemperatureExceeded), fps_improvement = +60.0 lifecycle : none - the candidate was never applied provider after : mock.value = 10 (untouched - nothing was applied) persisted experiment_trials rows: id 1, recorded_at 2026-07-27T10:45:45.919Z, payload 1499 bytes id 2, recorded_at 2026-07-27T10:45:45.920Z, payload 1416 bytes replay from a fresh handle (provider now at mock.value = 0, no spec in hand, no chat history): [promoted] trial 1: recorded Promote (Promoted) -> recomputed Promote (Promoted); identical = true, policy legal = true [rejected] trial 2: recorded Reject (TemperatureExceeded) -> recomputed Reject (TemperatureExceeded); identical = true, policy legal = true [tampered] trial 3: bounds widened to 200 C; recomputed verdict still identical = true, but policy legal = false (experiment spec rejected: max_temperature_c is 200, outside the 0 exclusive to 90 inclusive the policy allows) acceptance criterion: 2 measured experiments reached a decision through the immutable evaluator - Promote and Reject - and both re-evaluate identically from the journal alone

   Compiling fpsmaxxing-experiment-runner v0.1.0 (/home/fleet/.no-mistakes/worktrees/00b726bc730d/01KYHCCMW79GFCTH858G2ZM5QX/apps/experiment-runner)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running tests/acceptance_transcript.rs (target/debug/deps/acceptance_transcript-5483bf74be730dd7)

running 1 test
test one_measured_experiment_is_promoted_and_another_rejected_then_both_replay ... journal file: an on-disk SQLite database

provider parked at mock.value = 10

[promoted] trial id 1
  hypothesis      : raising mock.value from 10 to 40 improves FPS within thermal and power limits
  baseline value  : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples
  candidate value : 40 -> mean 100.0 fps, 70.0 C, 140.0 W, 0 errors over 5 samples
  bounds          : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples
  verdict         : Promote (Promoted), fps_improvement = +30.0
  lifecycle       : provider mock, preview "set mock.value from 10 to 40", verified = true, rolled_back = true
  provider after  : mock.value = 10 (the lease restored the pre-state)

[rejected] trial id 2
  hypothesis      : raising mock.value from 10 to 70 improves FPS within thermal and power limits
  baseline value  : 10 -> mean 70.0 fps, 55.0 C, 110.0 W, 0 errors over 5 samples
  candidate value : 70 -> mean 130.0 fps, 85.0 C, 170.0 W, 0 errors over 5 samples
  bounds          : >= 5.0 fps gain, <= 80.0 C, <= 200.0 W, <= 0 errors, >= 3 samples
  verdict         : Reject (TemperatureExceeded), fps_improvement = +60.0
  lifecycle       : none - the candidate was never applied
  provider after  : mock.value = 10 (untouched - nothing was applied)

persisted experiment_trials rows:
  id 1, recorded_at 2026-07-27T10:45:45.919Z, payload 1499 bytes
  id 2, recorded_at 2026-07-27T10:45:45.920Z, payload 1416 bytes

stored payload of trial 1:
  {"schema_version":1,"spec":{"hypothesis":"raising mock.value from 10 to 40 improves FPS within thermal and power limits","target":{"capability_id":"mock.value","parameters":{"value":40},"lease_seconds":30},"warmup_samples":2,"baseline_samples":5,"candidate_samples":5,"bounds":{"min_samples":3,"min_fps_improvement":5.0,"max_temperature_c":80.0,"max_power_w":200.0,"max_errors":0}},"baseline_value":10,"candidate_value":40,"baseline_samples":[{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0},{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0},{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0},{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0},{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0}],"candidate_samples":[{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0},{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0},{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0},{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0},{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0}],"verdict":{"decision":"promote","reason":"promoted","fps_improvement":30.0,"baseline":{"samples":5,"mean_fps":70.0,"max_temperature_c":55.0,"max_power_w":110.0,"total_errors":0},"candidate":{"samples":5,"mean_fps":100.0,"max_temperature_c":70.0,"max_power_w":140.0,"total_errors":0}},"lifecycle":{"provider_id":"mock","preview":"set mock.value from 10 to 40","verified":true,"rolled_back":true},"lifecycle_error":null}

replay from a fresh handle (provider now at mock.value = 0, no spec in hand, no chat history):
  [promoted] trial 1: recorded Promote (Promoted) -> recomputed Promote (Promoted); identical = true, policy legal = true
  [rejected] trial 2: recorded Reject (TemperatureExceeded) -> recomputed Reject (TemperatureExceeded); identical = true, policy legal = true
  [tampered] trial 3: bounds widened to 200 C; recomputed verdict still identical = true, but policy legal = false (experiment spec rejected: max_temperature_c is 200, outside the 0 exclusive to 90 inclusive the policy allows)

acceptance criterion: 2 measured experiments reached a decision through the immutable evaluator - Promote and Reject - and both re-evaluate identically from the journal alone
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
Evidence: Verbatim persisted TrialRecord payload of the promoted trial (read from SQLite)
{"schema_version":1,"spec":{"hypothesis":"raising mock.value from 10 to 40 improves FPS within thermal and power limits","target":{"capability_id":"mock.value","parameters":{"value":40},"lease_seconds":30},"warmup_samples":2,"baseline_samples":5,"candidate_samples":5,"bounds":{"min_samples":3,"min_fps_improvement":5.0,"max_temperature_c":80.0,"max_power_w":200.0,"max_errors":0}},"baseline_value":10,"candidate_value":40,"baseline_samples":[{"fps":70.0,"temperature_c":55.0,"power_w":110.0,"errors":0}, ... x5],"candidate_samples":[{"fps":100.0,"temperature_c":70.0,"power_w":140.0,"errors":0}, ... x5],"verdict":{"decision":"promote","reason":"promoted","fps_improvement":30.0,"baseline":{"samples":5,"mean_fps":70.0,"max_temperature_c":55.0,"max_power_w":110.0,"total_errors":0},"candidate":{"samples":5,"mean_fps":100.0,"max_temperature_c":70.0,"max_power_w":140.0,"total_errors":0}},"lifecycle":{"provider_id":"mock","preview":"set mock.value from 10 to 40","verified":true,"rolled_back":true},"lifecycle_error":null}
Evidence: Demo CLI run twice in separate processes - identical output proves determinism

$ cargo run -p fpsmaxxing-experiment-runner # run 1 trial 1 -> Promote (Promoted); fps_improvement = 30.0 lifecycle: provider mock, verified = true, rolled_back = true replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true exit=0 $ cargo run -p fpsmaxxing-experiment-runner # run 2 (separate process, no shared state) trial 1 -> Promote (Promoted); fps_improvement = 30.0 lifecycle: provider mock, verified = true, rolled_back = true replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true exit=0

$ cargo run -p fpsmaxxing-experiment-runner   # run 1
trial 1 -> Promote (Promoted); fps_improvement = 30.0
  lifecycle: provider mock, verified = true, rolled_back = true
replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true
exit=0

$ cargo run -p fpsmaxxing-experiment-runner   # run 2 (separate process, no shared state)
trial 1 -> Promote (Promoted); fps_improvement = 30.0
  lifecycle: provider mock, verified = true, rolled_back = true
replay 1 -> recomputed Promote; consistent with journal = true, policy legal = true
exit=0
Evidence: Named behaviour coverage: schema parity, evaluator known-answer fixtures, integration scenarios
$ cargo test -p fpsmaxxing-contracts   # typed spec/verdict <-> schemas/*.json parity
test tests::capability_fields_match_capability_schema ... ok
test tests::decision_bounds_are_bounded_like_the_schema ... ok
test tests::enum_wire_strings_match_capability_schema ... ok
test tests::experiment_and_verdict_wire_types_round_trip ... ok
test tests::experiment_types_reject_unknown_fields ... ok
test tests::generated_schemas_match_checked_in_schemas ... ok
test tests::lease_seconds_is_bounded_like_the_schema ... ok
test tests::lease_seconds_zero_is_rejected ... ok
test tests::manifest_fields_match_sidecar_schema ... ok
test tests::protocol_version_zero_is_rejected_like_the_schema ... ok
test tests::sample_counts_are_bounded_like_the_schema ... ok
test tests::the_hypothesis_is_bounded_like_the_schema ... ok
test tests::unknown_fields_are_rejected_like_the_schemas ... ok
test tests::verdict_enum_wire_strings_match_schema ... ok
test tests::wire_types_round_trip ... ok

$ cargo test -p fpsmaxxing-experiment-runner --lib   # evaluator known-answer fixtures, model, spec gate
test evaluator::tests::an_empty_set_reports_zero_throughout ... ok
test evaluator::tests::an_error_total_beyond_u64_saturates_and_rejects ... ok
test evaluator::tests::errors_ceiling_rejects_a_regressing_candidate ... ok
test evaluator::tests::evaluation_is_deterministic_across_repeated_calls ... ok
test evaluator::tests::improvement_exactly_at_threshold_promotes ... ok
test evaluator::tests::maxima_report_the_worst_observation_even_when_all_are_negative ... ok
test evaluator::tests::power_ceiling_rejects_even_with_a_large_gain ... ok
test evaluator::tests::promotes_when_every_bound_is_met ... ok
test evaluator::tests::rejects_when_a_set_is_below_minimum_samples ... ok
test evaluator::tests::rejects_when_improvement_is_below_threshold ... ok
test evaluator::tests::temperature_ceiling_is_checked_before_improvement ... ok
test model::tests::faults_appear_only_past_the_safe_onset ... ok
test model::tests::higher_values_raise_fps_temperature_and_power ... ok
test model::tests::measurement_is_deterministic ... ok
test model::tests::returns_exactly_the_counted_samples ... ok
test model::tests::warmup_samples_are_discarded ... ok
test runner::tests::accepts_a_spec_inside_the_envelope ... ok
test runner::tests::accepts_bounds_exactly_at_the_policy_envelope ... ok
test runner::tests::rejects_a_candidate_value_above_the_policy_bound ... ok
test runner::tests::rejects_a_capability_the_provider_does_not_advertise ... ok
test runner::tests::rejects_a_hypothesis_above_the_policy_bound ... ok
test runner::tests::rejects_a_lease_above_the_policy_bound ... ok
test runner::tests::rejects_a_target_without_an_unsigned_value ... ok
test runner::tests::rejects_an_advertised_capability_the_model_does_not_describe ... ok
test runner::tests::rejects_an_empty_hypothesis ... ok
test runner::tests::rejects_bounds_looser_than_the_policy_envelope ... ok
test runner::tests::rejects_counts_below_the_minimum_the_bounds_require ... ok
test runner::tests::rejects_sample_counts_above_the_ceiling ... ok
test runner::tests::rejects_target_parameters_the_model_does_not_take ... ok

$ cargo test -p fpsmaxxing-experiment-runner --test integration
test a_baseline_outside_the_policy_bound_is_refused_before_any_measurement ... ok
test a_candidate_outside_the_policy_bound_is_refused_before_any_measurement ... ok
test a_lease_outside_the_policy_bound_is_refused_before_any_measurement ... ok
test a_lost_record_reports_the_lifecycle_the_promotion_had_already_run ... ok
test a_promoted_experiment_runs_the_full_lifecycle ... ok
test a_promoted_trial_survives_a_lifecycle_the_broker_refuses ... ok
test a_refused_lifecycle_reports_why_its_trial_could_not_be_journaled ... ok
test a_rejected_experiment_is_never_applied_and_leaves_the_baseline ... ok
test a_replay_does_not_depend_on_the_attached_provider ... ok
test a_replay_reports_a_record_the_policy_gate_would_refuse ... ok
test a_replay_reports_bounds_outside_the_policy_envelope ... ok
test a_replay_reports_lifecycle_fields_that_contradict_the_verdict ... ok
test a_trial_record_carrying_unknown_fields_fails_closed ... ok
test a_trial_record_from_an_unsupported_version_fails_closed ... ok
test a_trial_record_without_a_readable_version_fails_closed ... ok
test a_trial_replays_from_the_journal_alone_with_an_identical_verdict ... ok
test a_trial_targeting_an_unadvertised_capability_is_refused ... ok
test an_out_of_envelope_spec_is_refused_before_any_measurement ... ok
test mvp_one_measured_experiment_is_promoted_or_rejected_by_the_evaluator ... ok
test replaying_a_trial_that_was_never_recorded_fails_closed ... ok
test target_parameters_the_model_does_not_take_are_refused_before_any_measurement ... ok
test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.17s
Evidence: Full workspace test run summary
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_broker-8b0053f98bd2f7ab)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_cli-411430e0f51723f6)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_contracts-a3cc4198a1c7056f)
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_control_plane-2a8898118987d608)
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_experiment_runner-e47cce48e658f156)
test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_experiment_runner-9773b3d0e5f4b728)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/acceptance_transcript.rs (target/debug/deps/acceptance_transcript-5483bf74be730dd7)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
     Running tests/integration.rs (target/debug/deps/integration-3a308fe69f3306cc)
test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_gateway-0923a1f81c9c68db)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_gateway-a705aa0bdae59449)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/mcp_e2e.rs (target/debug/deps/mcp_e2e-9a86ca7e6ccf5bb8)
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_mock_provider-06718f1f81aec59c)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_mock_provider-39bc58f6fad5ef72)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/lib.rs (target/debug/deps/fpsmaxxing_provider_sdk-f0e1c7f0e8a274b6)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/fpsmaxxing_watchdog-58c836a4580166be)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running unittests src/main.rs (target/debug/deps/xtask-f7b4cd0a2ed38e94)
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Pipeline

Updates from git push no-mistakes

... (3 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)

⚠️ **Review** - 2 infos

🔧 Fix: gate lease, lifecycle result fields, and record version
6 issues (1 warning, 5 infos) still open:

  • ⚠️ crates/control-plane/src/lib.rs:297 - The broker's lease ceiling is now enforced by an untested branch. a_promoted_trial_survives_a_lifecycle_the_broker_refuses (integration.rs:223) was the only test in the workspace that reached PolicyDenied(&#34;lease exceeds 300 seconds&#34;), and this commit correctly re-levered it to an ApprovalRequired risk class because the runner rejects an oversized lease before measurement now. Nothing replaced the lost coverage: crates/control-plane's own test module never had a lease case (verified against base 7f3ab1d), and the gateway e2e only covers the value bound (mcp_e2e.rs:164, value 101). The branch is still reachable - the gateway's fpsmaxxing.run_mock_lifecycle tool accepts lease_seconds straight from the LLM and hands it to run_lifecycle - so an off-by-one (&gt; vs &gt;=) or a change to MAX_LEASE_SECONDS would ship silently. The TTL lease is what bounds how long a mutation may persist, one of the repo's non-negotiable boundaries. Fix is mechanical: a unit test in the control-plane test module asserting run_lifecycle denies MAX_LEASE_SECONDS + 1 and accepts MAX_LEASE_SECONDS.
  • ℹ️ apps/gateway/src/lib.rs:67 - MAX_LEASE_SECONDS was introduced with the stated purpose that callers hold the lease "to the same ceiling this policy does rather than restating it", and MAX_MOCK_VALUE says the same. The gateway's tools/list response still restates both as literals: &#34;value&#34;: {&#34;maximum&#34;: 100} and &#34;lease_seconds&#34;: {&#34;maximum&#34;: 300}. That copy is the one the LLM reads to decide what to request, and no test pins it to the constants, so tightening policy would leave the advertised schema over-promising a value the broker then denies. apps/gateway already depends on fpsmaxxing-control-plane, so this is a mechanical substitution into the json! macro. (sidecars/mock-provider/src/lib.rs:43 also hardcodes maximum: 100, but a sidecar importing the control plane would invert the layering - leave that one.)
  • ℹ️ schemas/experiment.schema.json:31 - validate_spec now rejects lease_seconds &gt; 300, but the published spec schema declares {&#34;type&#34;: &#34;integer&#34;, &#34;minimum&#34;: 1} with no maximum, so an LLM authoring a spec against the contract can produce a schema-valid document the runner refuses with InvalidSpec. Every other policy bound on a typed spec field is mirrored there and pinned by a contract test - MAX_SAMPLES on the three sample counts, and each DecisionBounds threshold via decision_bounds_are_bounded_like_the_schema. MAX_MOCK_VALUE is unmirrored only because it lives inside the opaque parameters object; lease_seconds is a first-class typed field, so the same excuse does not apply. Mirroring it properly means deciding where the constant belongs: a #[schemars(range(max = ...))] on ChangeRequest.lease_seconds would move MAX_LEASE_SECONDS into crates/contracts and apply it to every ChangeRequest, not just experiment targets - which is a real design call, hence flagging rather than fixing.
  • ℹ️ docs/ARCHITECTURE.md:28 - Three places enumerate exactly what the replay policy gate re-checks, and none were updated when the lease bound joined it in the final commit: ARCHITECTURE.md:28 ("a row whose spec, capability, declared sample counts, candidate value, or baseline was rewritten"), ADR 0002 lines 27 and 30 ("capability, sample counts, decision bounds, and candidate value" / "the spec, the capability, the declared sample counts, the candidate value, the baseline ceiling, and the lifecycle fields"), and the main.rs:10-11 module doc that lists the same fields for the demo's exit-code contract. ARCHITECTURE.md:27 likewise lists what is bounded before measurement without the lease. These enumerations are the auditable description of the tamper gate, and a_replay_reports_a_record_the_policy_gate_would_refuse now has a lease case the docs do not describe.
  • ℹ️ apps/experiment-runner/src/runner.rs:429 - Noting the coupling rather than asking for a change: the new gate requires rolled_back == true on every recorded lifecycle, which is sound today because run_stages hard-codes rolled_back: true on its only Ok path (control-plane/src/lib.rs:347). But rolled_back means "the captured baseline was restored", and the documented next step is durable keep-or-rollback under the privileged broker - at which point a legitimately kept promotion carries rolled_back: false and this audit gate flags it as tampered with. Archived rows stay legal (they all carry true), so the migration is a gate relaxation rather than an audit break, but it is a forward-compat coupling worth knowing about when the keep path lands. Same for verified, though that one is a genuine invariant of observe_applied rather than an artifact.
  • ℹ️ apps/experiment-runner/src/runner.rs:287 - On the lifecycle-failure path run_trial journals the record, then returns Err(error) and drops journaled - the id of the row it just wrote. The doc promises the record is journaled before the error surfaces, but a caller cannot address it: recovering the id means scanning trial_ids() and assuming the last entry is theirs, which is what integration.rs:243 does via ids[0]. That assumption holds only single-threaded; two processes sharing one journal file (a supported configuration - begin_experiment takes an immediate transaction with a busy timeout specifically to make concurrent gateways safe) can interleave appends, and the recovering caller has no way to identify its own row. Carrying the id on the error variant, or returning Result&lt;StoredTrial, (i64, RunnerError)&gt;, would close it - a public signature change, so flagging rather than fixing.

🔧 Fix: test lease gate, share lease cap, carry trial id
5 infos still open:

  • ℹ️ apps/experiment-runner/src/runner.rs:118 - A TrialRecord carries no link to the lifecycle journal rows it authorized. run_lifecycle allocates a correlation id in begin_experiment (control-plane/src/lib.rs:396) and writes eight stage rows under it, but LifecycleResult (control-plane/src/lib.rs:97-105) does not return that id, so LifecycleOutcome cannot record it and no public API exposes it (journal_stages() returns stage names only). An auditor holding a trial row therefore cannot find the experiment_journal rows for the mutation it authorized, or vice versa, except by correlating recorded_at timestamps across the two tables - which is ambiguous in exactly the configuration the design explicitly supports: concurrent gateways sharing one journal file (ADR 0002 consequences). The crash story still works (a dangling apply-intent with no trial row is what doctor reports), so this is forward correlation only. Closing it means adding an experiment_id to LifecycleResult and mirroring it on LifecycleOutcome - a public control-plane type change, which the task deliberately kept additive and minimal, hence flagging rather than fixing.
  • ℹ️ apps/experiment-runner/src/runner.rs:304 - This commit restructured the lifecycle-failure path specifically to stop callers guessing at journal state, but the secondary failure on that path is still unstructured: when record_trial fails, the ControlPlaneError is eprintln!'d from a library crate and discarded, and the caller receives only trial_id: None. That is the more serious condition of the two - the measurements that authorized a mutation were lost entirely, not merely accompanied by a broker refusal - yet it is the only one a programmatic consumer cannot inspect: a disk-full journal, a serialization fault, and a locked database are indistinguishable. The precedence rule (lifecycle error wins) is right and documented; carrying the journal error alongside it, rather than only to stderr, would finish what the trial_id change started. Public error-type change, so flagging rather than fixing.
  • ℹ️ apps/experiment-runner/src/runner.rs:406 - check_policy re-runs the current validate_spec over a journaled spec, so tightening any policy constant retroactively marks every archived row recorded under the looser ceiling as policy_legal = false - reported to an auditor as "the journaled row is not one this runner would have written" and exiting the demo non-zero (main.rs:76). Lowering MAX_DECISION_TEMPERATURE_C from 90 to 85 would flag every historical trial with max_temperature_c: 88; the same holds for MAX_LEASE_SECONDS, MAX_MOCK_VALUE, and MAX_SAMPLES, and the lease joining the gate in this branch widens the surface. TRIAL_RECORD_VERSION does not cover it: a constant change is not a record-format change, so nothing forces a bump. This is the same false-alarm class the authors already reasoned about and rejected for the provider manifest (runner.rs:375-380, ADR 0002 final rationale line), left unaddressed for the constants themselves. Options are to journal the envelope with the record, or to document the constant-drift case next to the manifest paragraph in ADR 0002 and treat a constant change as a version bump - a real design call.
  • ℹ️ crates/contracts/src/lib.rs:205 - ExperimentSpec.hypothesis is an unbounded String written verbatim into a single durable journal row by record_trial. Every other LLM-authored field on the spec carries a ceiling declared in this crate and mirrored in schemas/experiment.schema.json - the three sample counts against MAX_SAMPLES, each DecisionBounds threshold, and now lease_seconds - and the schema gives hypothesis a minLength: 1 but no maximum. There is no LLM-reachable path to run_trial today (the gateway exposes only run_mock_lifecycle), so this is a latent rather than live concern, but the spec is designed to be agent-authored and the row size is whatever the author sends. A maxLength in the schema plus a matching check in validate_spec would follow the pattern the sample counts already set; choosing the limit is a product decision.
  • ℹ️ crates/contracts/src/lib.rs:71 - The new MAX_LEASE_SECONDS doc enumerates where the ceiling is mirrored - schemas/experiment.schema.json, the broker policy, the experiment runner - and omits the fourth site: apps/gateway/src/lib.rs:67 restates &#34;lease_seconds&#34;: { &#34;maximum&#34;: 300 } as a literal in the tools/list schema an LLM reads to decide what to request. The same commit added exactly that caveat to MAX_MOCK_VALUE ("Callers that publish their own request schema, such as the gateway's advertised tool input, state the bound independently", control-plane/src/lib.rs:19-20), so the asymmetry leaves the lease's known drift risk with no in-code pointer: tightening the constant would leave the advertised schema over-promising a value the broker then denies, and no test pins the two together. One sentence mirroring the MAX_MOCK_VALUE caveat; no gateway edit needed.

🔧 Fix: carry journal error, bound hypothesis, document policy drift
4 issues (1 warning, 3 infos) still open:

  • ⚠️ docs/adr/0002-alpha-experiment-journal.md:35 - The new policy-drift paragraph prescribes a remedy that destroys the signal it exists to preserve. Lines 33-34 establish that policy_legal reports a row against the policy in force now, so tightening MAX_LEASE_SECONDS, MAX_MOCK_VALUE, MAX_SAMPLES, MAX_HYPOTHESIS_CHARS, or a decision-bound ceiling deliberately flags every archived row recorded under the looser one - "an auditor wants them surfaced rather than grandfathered". Line 35 then makes a policy-constant change a TRIAL_RECORD_VERSION bump "so a flagged archive is attributable to the constant that moved". But the version gate is fail-closed and runs first: replay_trial (runner.rs:355-362) reads schema_version off the raw payload and returns UnsupportedRecordVersion before check_policy is ever called, so after the bump every pre-existing row errors out of replay instead of replaying with policy_legal = false. Nothing is flagged - the whole archive becomes unreadable, and main.rs:69 propagates that as a hard Err rather than the non-zero policy-gate exit at main.rs:84. The same claim is repeated in the check_policy doc (runner.rs:390-391). The two mechanisms are also in tension by design: the version is documented at ADR line 26 as tracking "what a row contains" (a field addition a reader must refuse), while line 35 explicitly says a constant change is "a change in what the journal means rather than in what a row contains" - and then routes it through the refusal mechanism anyway. Either the constant drift needs its own attribution (journaling the envelope with the record, or a separate policy-generation field the gate reads without refusing the row), or the ADR should state plainly that a constant bump retires the existing archive from replay.
  • ℹ️ apps/experiment-runner/src/runner.rs:384 - Four enumerations now list the hypothesis among the fields whose rewrite replay detects: check_policy's doc ("a row whose capability, hypothesis, sample counts, decision bounds, candidate value, or TTL lease were rewritten is reported as illegal"), ARCHITECTURE.md:28, ADR 0002 lines 27 and 30, and the main.rs:11 module doc that defines the demo's exit-code contract. Only the length ceiling is actually checked (runner.rs:557-562). Every other field in those lists has a second term to check against - the capability has exactly one legal value, candidate_value and the sample counts are cross-checked against redundant copies in the record, the lease and bounds have policy ceilings that a rewrite must clear - so an arbitrary rewrite of any of them trips the gate. The hypothesis has none: it is free text with no redundant copy, so rewriting "raise mock.value to 40" to "lower mock.value to 40" replays as policy_legal = true. That is the field a tamperer would most want to rewrite, because it is the human-readable statement of what a promotion was for, and the docs now assert it is protected when only its length is. The fix is wording: say the hypothesis is bounded rather than that its rewrite is detected, and keep the ADR:30 "detection is structural only" caveat honest about which structure exists.
  • ℹ️ apps/experiment-runner/src/runner.rs:557 - schemas/experiment.schema.json:16 declares &#34;hypothesis&#34;: { &#34;type&#34;: &#34;string&#34;, &#34;minLength&#34;: 1, &#34;maxLength&#34;: 4096 }, but only the ceiling is enforced in Rust. validate_spec checks chars().count() &gt; MAX_HYPOTHESIS_CHARS and nothing else, and #[schemars(length(max = MAX_HYPOTHESIS_CHARS))] (contracts/src/lib.rs:217) emits no minLength, so the generated schema and the checked-in one disagree on the floor - the parity test only compares property names, required, and additionalProperties, so it does not catch the difference, and the_hypothesis_is_bounded_like_the_schema pins only maxLength. Consequences: a spec with hypothesis: &#34;&#34; is accepted, measured, and journaled despite the published contract forbidding it, and on replay a row whose hypothesis was blanked out passes the gate that now refuses one that was lengthened. Blanking is the cheaper tamper of the two. Adding length(min = 1) plus a matching check would close both halves; it is a small behavior change on a wire contract, so it is worth confirming rather than applying silently.
  • ℹ️ apps/experiment-runner/src/runner.rs:326 - This commit made the lifecycle-failure path fully inspectable - LifecycleFailed now carries the trial id or the reason the record was lost - but the mirror case is untouched. When a promotion's lifecycle succeeds and record_trial then fails, journaled? returns a bare RunnerError::ControlPlane(ControlPlaneError::Journal(_)) with no indication that a lifecycle ran at all. That is the case where the provider was actually mutated (apply, verify, rollback all completed) and the measurements authorizing it were lost, yet the caller learns strictly less than in the refused-lifecycle case where nothing reached the provider. doctor cannot recover it either: it flags experiments with an apply-intent and no terminal record (ADR 0002 consequences), and this experiment has a completed record, so a completed lifecycle with no trial row is invisible to both readers. Today the state impact is nil because run_lifecycle always restores the pre-state, but the documented next step is durable keep-or-rollback under the privileged broker - at which point this is a knob left mutated with no trial record and no dangling-intent flag. Carrying the lifecycle outcome on the error, the way the failure path now carries the journal error, would close it; that is another public error-type change, hence flagging rather than fixing.

🔧 Fix: fix policy-drift docs, hypothesis floor, and lost-record lifecycle
4 infos still open:

  • ℹ️ apps/experiment-runner/src/runner.rs:639 - With the hypothesis now bounded at both ends, spec.target.parameters is the field that inherits the problem the hypothesis bound was added to solve. validate_spec reads exactly one key out of it (candidate_value -> parameters[&#34;value&#34;], runner.rs:726-732) and bounds that against MAX_MOCK_VALUE; every other key passes through unexamined. parameters is a bare serde_json::Value (contracts/src/lib.rs:86) whose schema is {&#34;type&#34;: &#34;object&#34;} with no maxProperties, maxLength, or depth limit, so {&#34;value&#34;: 40, &#34;pad&#34;: &#34;&lt;10 MB&gt;&#34;} is accepted, measured, and written verbatim - and written twice durably, once into the trial row by record_trial and again into the lifecycle journal's apply-intent record by run_stages (control-plane/src/lib.rs:324), since run_lifecycle serializes the whole ChangeRequest. validate_spec's own doc (runner.rs:572-573) states "Everything the measurement phase and the decision gate consume arrives over the wire, so all of it is bounded before any measurement work runs", which is now true of every spec field except this one. Reachability matches the hypothesis case exactly: latent, not live - the gateway builds parameters itself from a validated integer (gateway/src/lib.rs:90-91), so there is no LLM-reachable path to an arbitrary object today, but ExperimentSpec is designed to be agent-authored. Closing it means bounding a field on the shared ChangeRequest (a size cap, or requiring parameters hold only the keys the capability's input_schema declares), which affects the broker path too - a design call rather than a mechanical fix.
  • ℹ️ apps/experiment-runner/src/runner.rs:607 - The floor is spec.hypothesis.chars().count() &lt; 1, so it catches exactly the empty string. Its stated purpose is that "a promotion cannot be journaled with no statement of what it was for" (runner.rs:590-591) and, on the replay side, that "blanking the statement of what a promotion was for is the cheaper rewrite of the two" (integration.rs:683-686). A single space defeats both: hypothesis: &#34; &#34; is one character, passes validate_spec, passes check_policy, and carries exactly as much meaning as &#34;&#34;. This matters more on the replay side than the run-time side - the run-time caller has no motive to send a space, but a tamperer rewriting a journaled row to erase what a promotion was for now simply writes &#34; &#34; instead of &#34;&#34; and the gate that was just added to catch that reports policy_legal = true. The checked-in schema has the same hole (minLength: 1 admits &#34; &#34;), so keeping the two aligned means either adding a &#34;pattern&#34;: &#34;\\S&#34; alongside a trim().is_empty() check, or accepting the limit and saying in the doc that the floor bounds length rather than substance - the same distinction this commit just drew for the ceiling.
  • ℹ️ apps/experiment-runner/src/runner.rs:449 - Three places now justify not detecting a sample rewrite by saying it would need an external anchor: check_policy's doc ("catching that needs each row anchored outside itself - a signed or hash-chained journal"), ARCHITECTURE.md:29, and ADR 0002 line 32. That reason does not hold for the alpha as built. model::measure is a pure function of (value, warmup, counted) (model.rs:52-62) and every one of those three inputs is already in the record: baseline_value/candidate_value and spec.warmup_samples plus the two declared counts. So record.baseline_samples == model::measure(record.baseline_value, record.spec.warmup_samples, record.spec.baseline_samples.get()) is an invariant of every row this runner wrote, checkable with no anchor at all - the same shape as the count check holds_declared_count already performs one field over. A coherent rewrite of the measurements plus the verdict, which the docs name as the one surviving tamper, would be caught outright. Whether to add it is a genuine call: replay already consults MODELED_CAPABILITY_ID, but re-deriving contents couples the audit gate to the measurement model in a way that must be deleted the moment real PresentMon telemetry lands (real samples are not reproducible, which is the whole reason they are journaled verbatim). Either add the check while the model is pure, or correct the stated reason from "needs a hash chain" to "deliberately not coupled to the measurement model, so the alpha does not exploit its reproducibility".
  • ℹ️ apps/experiment-runner/src/main.rs:55 - TrialNotJournaled was added specifically so a caller learns that a promotion reached the provider and its record was lost, but the demo - the one caller in the tree - falls through to Err(error) =&gt; return Err(error), which Termination renders as a derived Debug dump rather than the crafted Display ("trial record was lost after its promotion completed the lifecycle"). The sibling LifecycleFailed gets nine lines of tailored reporting immediately above (main.rs:36-54). The module doc at main.rs:14-17 still enumerates three failure modes and says the demo reports how a lost row was lost, which is now the case for one of the two lost-record paths. Exit status is unaffected - returning Err from main is non-zero either way - and the demo's :memory: journal makes the path unreachable in practice, so this is about the enumeration matching the code rather than a behavioral gap. A fourth match arm printing source and whether a lifecycle ran, plus a sentence in the module doc, closes it.

🔧 Fix: bound target parameters, correct sample-integrity docs, report lost record
2 issues (1 warning, 1 info) still open:

  • ⚠️ apps/experiment-runner/src/runner.rs:451 - The replay gate's four enumerations claim more detection than check_policy performs, and the claim is load-bearing because it is the auditable description of what a clean replay proves.

check_policy cross-checks record.candidate_value against the value it re-derives from record.spec, and bounds record.baseline_value by MAX_MOCK_VALUE. Neither is tied to the recorded samples. Two consequences:

(1) Rewriting spec.target.parameters.value and candidate_value together to any other in-bounds value is undetected. Set both to 100 on a trial actually measured at 40: validate_spec returns 100, 100 == record.candidate_value, the samples and verdict are untouched so is_consistent() holds, and the row replays with policy_legal = true. The archive then states that value 100 was measured at 70 C and promoted, when those are the value-40 samples. This needs no sample forgery at all - two field edits.

(2) baseline_value has no second term anywhere; only the ceiling. Rewriting it from 10 to 90 passes replay outright, yet ARCHITECTURE.md:28 lists "or baseline was rewritten after the fact is reported as outside policy". (main.rs:11 and the ADR correctly say "baseline ceiling" - it is ARCHITECTURE.md that over-claims.)

The sentence to fix is runner.rs:451-452: "every other checked field has a second term to disagree with, while the hypothesis is free text with no redundant copy in the record." That is false for baseline_value, and for candidate_value the second term is a copy of the same claim rather than an independent one - so a coherent edit of the pair survives. The caveat paragraph names the surviving tamper as "a rewrite of the measurements together with the verdict they imply, or of the hypothesis text within its bounds", which understates it: relabelling the knob value requires neither.

This is the same class as the hypothesis wording corrected in round 4, and it is exactly what the sample re-derivation check would have caught - the round-5 decision not to re-derive is sound (real telemetry is not reproducible), but the docs should stop describing the resulting gap as covered. Note lifecycle.preview does embed the applied value for a promoted row and is unchecked, but it is provider-authored free text, so parsing it is not a serious cross-check.

Suggested fix is wording only, in all four places: say the candidate value is cross-checked against its own spec and the baseline is bounded, and extend the surviving-tamper caveat to name a coherent relabel of the candidate value and a rewrite of the baseline within its ceiling.

  • ℹ️ apps/experiment-runner/src/model.rs:29 - MODELED_PARAMETERS = [&#34;value&#34;] is now the runner's authority on which target parameters are acceptable, but no contract an agent can read says so, and the same fact is already declared elsewhere.

sidecars/mock-provider/src/lib.rs:41-45 advertises input_schema: {&#34;type&#34;: &#34;object&#34;, &#34;required&#34;: [&#34;value&#34;], &#34;properties&#34;: {&#34;value&#34;: {&#34;type&#34;: &#34;integer&#34;, &#34;minimum&#34;: 0, &#34;maximum&#34;: 100}}} - the capability's own published statement of what it takes - and it omits additionalProperties: false. schemas/experiment.schema.json:30 declares parameters as a bare {&#34;type&#34;: &#34;object&#34;}. So both published surfaces permit extra keys while validate_spec refuses them with InvalidSpec. An agent authoring a spec against either contract can produce a schema-valid document the runner rejects, which is the same divergence the round-2 lease_seconds finding closed for a typed field.

Nothing validates against input_schema at runtime (verified: it is only read by mcp_e2e.rs:84 for an assertion), so this is a contract-consistency issue rather than a live bug, and there is no LLM-reachable path to run_trial today.

The narrow fix that respects the round-5 instruction not to touch the shared ChangeRequest schema or the broker path is to add &#34;additionalProperties&#34;: false to the mock capability's input_schema, keeping the declaration with the capability that owns it. That leaves MODELED_PARAMETERS as a deliberate second copy - justified, since replay must not consult the attached provider's manifest (a_replay_does_not_depend_on_the_attached_provider) - but at least the published contract would agree with the runner. Flagging rather than fixing because it tightens a published sidecar contract.

🔧 Fix: correct replay-gate docs on candidate and baseline detection
2 infos still open:

  • ℹ️ apps/experiment-runner/src/runner.rs:448 - The enumerations rewritten in the final commit claim exhaustiveness they do not have. check_policy inspects exactly two of the recorded lifecycle's four fields (verified, rolled_back, runner.rs:528-535) and none of lifecycle_error's two. So runner.rs:448 - "A recorded lifecycle is held to its own contents too" - over-claims, and the "three rewrites" list at runner.rs:456-464, ARCHITECTURE.md:30, ADR 0002:31, and main.rs:15-19 misses a fourth class.

Concretely: on a promoted row, setting lifecycle.provider_id to any other string, or rewriting lifecycle.preview, decodes cleanly (LifecycleOutcome types them as free-text String), leaves the presence and verified &amp;&amp; rolled_back checks satisfied, leaves the verdict untouched, and replays as policy_legal = true. Same for a refused promotion: lifecycle_error.kind can be moved from "policy-denied" to "provider" and lifecycle_error.error rewritten wholesale, with nothing to contradict it.

This matters on the terms the ADR itself sets at line 29 - "the trial row is the only auditable statement that a promotion reached the provider". Which provider owned the mutation, and why the broker refused one, are part of that statement, and both are rewritable. Like the candidate-value gap corrected in the same commit, it cannot be closed by a second term: provider_id could only be cross-checked against the attached manifest, which a_replay_does_not_depend_on_the_attached_provider deliberately forbids.

Fix is wording in the same four places: say the recorded lifecycle is held to verified and rolled_back rather than to "its own contents", and add the lifecycle's provider and preview text plus the whole recorded broker error to the list of rewrites that survive.

  • ℹ️ docs/README.md:19 - CLAUDE.md directs every agent session to start at docs/README.md, and its one-line summary for ADR 0002 still reads "Write-ahead apply intent, terminal outcomes, and deferred two-phase journaling". This branch grew that ADR by ~20 lines covering the experiment_trials table, the append-only trial-record contract, the replay tamper gate and exactly what it detects, and the policy-constant drift rule - and it explicitly supersedes one of the deferrals the blurb advertises ("This supersedes the original deferral of a dedicated experiments table", ADR line 13). A reader scanning the index for where the trial-journal and replay-gate decisions are recorded would not find them. One clause added to the blurb.
✅ **Test** - passed

✅ No issues found.

  • cargo test --workspace (all suites green: 15 contracts, 14 control-plane, 29 experiment-runner unit, 21 experiment-runner integration, 2 gateway mcp_e2e, 1 mock-provider)
  • cargo run -p fpsmaxxing-experiment-runner run twice in separate processes - byte-identical output, confirming the evaluator/model are deterministic and wall-clock-free
  • cargo test -p fpsmaxxing-experiment-runner --test acceptance_transcript -- --nocapture (new transcript test I added: promote -> reject -> persisted-journal dump -> journal-only replay -> tampered-row gate)
  • cargo test -p fpsmaxxing-experiment-runner --test integration (named coverage incl. mvp_one_measured_experiment_is_promoted_or_rejected_by_the_evaluator, a_promoted_experiment_runs_the_full_lifecycle, a_rejected_experiment_is_never_applied_and_leaves_the_baseline, a_trial_replays_from_the_journal_alone_with_an_identical_verdict)
  • cargo test -p fpsmaxxing-experiment-runner --lib (evaluator known-answer fixtures for each ordered threshold, measurement-model determinism, spec/policy-envelope gate)
  • cargo test -p fpsmaxxing-contracts (typed ExperimentSpec/Verdict/MetricSample vs schemas/*.json field, required-set, bound, and enum wire-string parity)
  • Manual: read the persisted experiment_trials rows with a plain rusqlite::Connection on the journal file to show durable state rather than describe it
  • Environment fix: built with throwaway cc/ar zig shims in the gitignored target/ dir (global cargo config pins CC=zig-cc, which rejects cc-rs's 4-component triple), then removed them
⚠️ **Document** - 1 info
  • ℹ️ docs/IMPLEMENTATION_PLAN.md:126 - The plan's MVP acceptance criteria and sprint phase exit gates have no convention for marking a criterion as satisfied, so the now-demonstrated criterion 'One measured experiment is promoted or rejected using the immutable evaluator' (and the 'Experiment engine' phase gate 'Trial replays without chat history') still read as pending. Introducing a status marker convention is an editorial decision for a human; I documented the implemented behavior in the Experiment loop section instead.
✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

undeemed added 16 commits July 24, 2026 06:10
…aluator and journal replay

Add the experiment engine phase: typed experiment contracts, a pure immutable
evaluator, a deterministic measurement model, and a trial runner that measures
baseline and candidate, gates them through the evaluator, and journals each
trial as a self-describing record that replays and re-evaluates from the
journal alone.

- contracts: experiment spec, decision bounds, metric sample/summary, and
  verdict wire types, with schemas/{experiment,verdict}.schema.json kept in
  sync (contract tests enforce field and enum parity).
- control-plane: additive read/journal APIs (snapshot, record_trial,
  read_trial, trial_ids) over a new experiment_trials table; no change to the
  existing journal schema or lifecycle.
- experiment-runner: immutable evaluator, deterministic model, trial runner,
  and replay; integration tests for a promoted trial, a rejected-and-preserved
  trial, journal-only replay reproducing the verdict, and the MVP acceptance
  criterion.

Safe-alpha limitation: mock capabilities are leased and the broker lifecycle
always rolls back, so the verdict gates whether the candidate is applied at
all rather than whether it persists; durable keep-or-rollback awaits the
privileged broker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant