Skip to content

fix(validation): evidence from a simulation loses its link to the run - #416

Merged
0xmanhnv merged 1 commit into
developfrom
fix/evidence-loses-simulation-link
Aug 4, 2026
Merged

fix(validation): evidence from a simulation loses its link to the run#416
0xmanhnv merged 1 commit into
developfrom
fix/evidence-loses-simulation-link

Conversation

@0xmanhnv

@0xmanhnv 0xmanhnv commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

triggerValidationEvidence passed nil for the simulation run id, so every validation_evidence row written by the agent-completes-command path landed with simulation_run_id NULL.

Live database:

 executor_kind | technique |   outcome    | sim_null
---------------+-----------+--------------+----------
 safe-check    | T1046     | not_detected | t
 safe-check    | T1046     | not_detected | t
 safe-check    | T1046     | not_detected | t
 safe-check    | T1046     | not_detected | t
 safe-check    | T1046     | not_detected | t

All 5 rows, all safe-checkevery one produced BY a simulation, and none traceable back to one. The field is exposed on the evidence API, so "which evidence did this run produce?" has always answered empty.

The value was never missing

payload.SimulationRunID is the same field the sibling function in the same file already reads:

// triggerSimulationFinalize — works, uses it to decide which run to finalize
if err := json.Unmarshal(cmd.Payload, &payload); err != nil || payload.SimulationRunID == "" {
    return
}

// triggerValidationEvidence — 60 lines below, same payload in hand
h.validationIngest.Ingest(bgCtx, tenantID, findingID, nil, ev)
                                                      ^^^

Two paths side by side, disagreeing about whether the field exists. The comment on the first one even says it "leaves the finding-evidence path entirely untouched" — accurate, and that was the bug.

Taking it from the command payload rather than from the agents submission also means the two paths now agree by construction, instead of depending on an agent remembering to echo the id back. (The other Ingest caller, validation_handler.go, does read it from the request — that path could always set it. This one is the path all 5 live rows came through.)

An unparseable id logs and drops the link, not the evidence. Losing a validation result would turn a provenance problem into a correctness one.

Why no test caught it

func (c *captureIngester) Ingest(_ context.Context, tenantID, findingID shared.ID, _ *shared.ID, ev validation.Evidence)
                                                                                  ^^^^^^^^^^^^

The mock discarded the argument. A mock that ignores a parameter cannot fail when that parameter is wrong. It now captures it.

New tests cover the link, the no-simulation case (a fabricated link is worse than an absent one), and the malformed case (evidence survives, verdict unaffected).

Verification

Proven fail-before/pass-after by restoring the nil — and by removing the whole block rather than just flipping the argument, since leaving an unused variable only produces a build error and proves nothing:

--- FAIL: TestTriggerValidationEvidence_CarriesTheSimulationLink
    evidence was ingested with a nil simulation run id even though the command
    payload carries one. The row lands with simulation_run_id NULL and the run
    it came from can never be traced to it
  • GOWORK=off go build ./... — ok
  • GOWORK=off go test ./... against app_test — all green
  • GOWORK=off make lint-ci — clean

Not claimed: the 5 existing rows are not backfilled. Their commands are long gone, so the link cannot be reconstructed — and inventing one would be exactly the kind of fabricated provenance this fix exists to avoid.

triggerValidationEvidence passed nil for the simulation run id, so every
validation_evidence row written by the agent-completes-command path landed with
simulation_run_id NULL.

On the live database that is all 5 rows, and all 5 are executor_kind=safe-check
— every one of them produced BY a simulation, and none traceable back to one.
The field is exposed on the evidence API (validation_handler.go), so "which
evidence did this run produce?" has always answered empty.

The value was never missing. payload.SimulationRunID is the same field the
sibling triggerSimulationFinalize already reads to decide which run to
finalize; the two paths sitting side by side in this file simply disagreed
about whether it existed. Taking it from the command payload rather than from
the agent's submission means they now agree by construction, instead of
depending on an agent remembering to echo the id back.

An unparseable id logs and drops the link rather than the evidence: losing a
validation result would turn a provenance problem into a correctness one.

The existing mock declared the parameter as `_ *shared.ID` — it discarded the
argument, which is why no test could ever have caught this. A mock that ignores
a parameter cannot fail when that parameter is wrong. It now captures it, and
the new tests cover the link, the no-simulation case (a fabricated link would
be worse than an absent one) and the malformed case.

Verified fail-before/pass-after by restoring the nil:
"evidence was ingested with a nil simulation run id even though the command
payload carries one".
@0xmanhnv
0xmanhnv merged commit 0a130b1 into develop Aug 4, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant