fix(validation): evidence from a simulation loses its link to the run - #416
Merged
Conversation
triggerValidationEvidence passed nil for the simulation run id, so every validation_evidence row written by the agent-completes-command path landed with simulation_run_id NULL. On the live database that is all 5 rows, and all 5 are executor_kind=safe-check — every one of them produced BY a simulation, and none traceable back to one. The field is exposed on the evidence API (validation_handler.go), so "which evidence did this run produce?" has always answered empty. The value was never missing. payload.SimulationRunID is the same field the sibling triggerSimulationFinalize already reads to decide which run to finalize; the two paths sitting side by side in this file simply disagreed about whether it existed. Taking it from the command payload rather than from the agent's submission means they now agree by construction, instead of depending on an agent remembering to echo the id back. An unparseable id logs and drops the link rather than the evidence: losing a validation result would turn a provenance problem into a correctness one. The existing mock declared the parameter as `_ *shared.ID` — it discarded the argument, which is why no test could ever have caught this. A mock that ignores a parameter cannot fail when that parameter is wrong. It now captures it, and the new tests cover the link, the no-simulation case (a fabricated link would be worse than an absent one) and the malformed case. Verified fail-before/pass-after by restoring the nil: "evidence was ingested with a nil simulation run id even though the command payload carries one".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
triggerValidationEvidencepassednilfor the simulation run id, so everyvalidation_evidencerow written by the agent-completes-command path landed withsimulation_run_id NULL.Live database:
All 5 rows, all
safe-check— every one produced BY a simulation, and none traceable back to one. The field is exposed on the evidence API, so "which evidence did this run produce?" has always answered empty.The value was never missing
payload.SimulationRunIDis the same field the sibling function in the same file already reads:Two paths side by side, disagreeing about whether the field exists. The comment on the first one even says it "leaves the finding-evidence path entirely untouched" — accurate, and that was the bug.
Taking it from the command payload rather than from the agents submission also means the two paths now agree by construction, instead of depending on an agent remembering to echo the id back. (The other
Ingestcaller,validation_handler.go, does read it from the request — that path could always set it. This one is the path all 5 live rows came through.)An unparseable id logs and drops the link, not the evidence. Losing a validation result would turn a provenance problem into a correctness one.
Why no test caught it
The mock discarded the argument. A mock that ignores a parameter cannot fail when that parameter is wrong. It now captures it.
New tests cover the link, the no-simulation case (a fabricated link is worse than an absent one), and the malformed case (evidence survives, verdict unaffected).
Verification
Proven fail-before/pass-after by restoring the
nil— and by removing the whole block rather than just flipping the argument, since leaving an unused variable only produces a build error and proves nothing:GOWORK=off go build ./...— okGOWORK=off go test ./...againstapp_test— all greenGOWORK=off make lint-ci— cleanNot claimed: the 5 existing rows are not backfilled. Their commands are long gone, so the link cannot be reconstructed — and inventing one would be exactly the kind of fabricated provenance this fix exists to avoid.