Skip to content

tests: validate every results submission and cross-check against signed vectors - #3

Open
laxsharma wants to merge 1 commit into
a2a-settlement:mainfrom
laxsharma:fix/validate-all-results
Open

tests: validate every results submission and cross-check against signed vectors#3
laxsharma wants to merge 1 commit into
a2a-settlement:mainfrom
laxsharma:fix/validate-all-results

Conversation

@laxsharma

Copy link
Copy Markdown
Contributor

Validate every results submission, and cross-check against the signed vector

tests/test_results.py hard-codes
RESULTS_PATH = results / "a2a-se" / "results.json", so any other
submission directory is not validated at all, and the checks it does run
are enum membership plus key presence.

Two consequences worth closing before the track takes more submissions:

  1. A submission can restate a vector's expected_verdict. Downgrading
    BLOCK to REVIEW currently validates, which quietly changes what a
    PASS means on the scoreboard.
  2. assert "observed" in entry is satisfied by "observed": null, so a
    PASS needs no evidence of what the rail actually did.

Changes:

  • Iterate results/*/results.json rather than one hard-coded path, with
    the rail id as the pytest parameter id so failures name the submission.
  • Cross-check each entry's expected_verdict against the signed vector's,
    which is authoritative.
  • Reject vector_id values that are not in the signed catalogue.
  • N/A requires a non-empty reason; PASS and FAIL require a
    non-empty observed object.

results/a2a-se passes unchanged. To confirm the checks bite, a scratch
submission with every verdict downgraded to REVIEW and
"observed": null fails on the verdict cross-check, where previously it
would not have been examined at all.

Verified locally: pytest tests/ is 15 passed.

This does not address self-attestation itself, which is a governance
question rather than a test one, but it does mean a submission has to be
wrong on purpose rather than by accident.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

… vector

`tests/test_results.py` hard-codes
`RESULTS_PATH = results / "a2a-se" / "results.json"`, so any other
submission directory is not validated at all, and the checks it does run
are enum membership plus key presence.

Two consequences worth closing before the track takes more submissions:

1. A submission can restate a vector's `expected_verdict`. Downgrading
   `BLOCK` to `REVIEW` currently validates, which quietly changes what a
   `PASS` means on the scoreboard.
2. `assert "observed" in entry` is satisfied by `"observed": null`, so a
   `PASS` needs no evidence of what the rail actually did.

Changes:

- Iterate `results/*/results.json` rather than one hard-coded path, with
  the rail id as the pytest parameter id so failures name the submission.
- Cross-check each entry's `expected_verdict` against the signed vector's,
  which is authoritative.
- Reject `vector_id` values that are not in the signed catalogue.
- `N/A` requires a non-empty `reason`; `PASS` and `FAIL` require a
  non-empty `observed` object.

`results/a2a-se` passes unchanged. To confirm the checks bite, a scratch
submission with every verdict downgraded to `REVIEW` and
`"observed": null` fails on the verdict cross-check, where previously it
would not have been examined at all.

Verified locally: `pytest tests/` is 15 passed.

This does not address self-attestation itself, which is a governance
question rather than a test one, but it does mean a submission has to be
wrong on purpose rather than by accident.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant