tests: validate every results submission and cross-check against signed vectors - #3
Open
laxsharma wants to merge 1 commit into
Open
tests: validate every results submission and cross-check against signed vectors#3laxsharma wants to merge 1 commit into
laxsharma wants to merge 1 commit into
Conversation
… vector `tests/test_results.py` hard-codes `RESULTS_PATH = results / "a2a-se" / "results.json"`, so any other submission directory is not validated at all, and the checks it does run are enum membership plus key presence. Two consequences worth closing before the track takes more submissions: 1. A submission can restate a vector's `expected_verdict`. Downgrading `BLOCK` to `REVIEW` currently validates, which quietly changes what a `PASS` means on the scoreboard. 2. `assert "observed" in entry` is satisfied by `"observed": null`, so a `PASS` needs no evidence of what the rail actually did. Changes: - Iterate `results/*/results.json` rather than one hard-coded path, with the rail id as the pytest parameter id so failures name the submission. - Cross-check each entry's `expected_verdict` against the signed vector's, which is authoritative. - Reject `vector_id` values that are not in the signed catalogue. - `N/A` requires a non-empty `reason`; `PASS` and `FAIL` require a non-empty `observed` object. `results/a2a-se` passes unchanged. To confirm the checks bite, a scratch submission with every verdict downgraded to `REVIEW` and `"observed": null` fails on the verdict cross-check, where previously it would not have been examined at all. Verified locally: `pytest tests/` is 15 passed. This does not address self-attestation itself, which is a governance question rather than a test one, but it does mean a submission has to be wrong on purpose rather than by accident. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validate every results submission, and cross-check against the signed vector
tests/test_results.pyhard-codesRESULTS_PATH = results / "a2a-se" / "results.json", so any othersubmission directory is not validated at all, and the checks it does run
are enum membership plus key presence.
Two consequences worth closing before the track takes more submissions:
expected_verdict. DowngradingBLOCKtoREVIEWcurrently validates, which quietly changes what aPASSmeans on the scoreboard.assert "observed" in entryis satisfied by"observed": null, so aPASSneeds no evidence of what the rail actually did.Changes:
results/*/results.jsonrather than one hard-coded path, withthe rail id as the pytest parameter id so failures name the submission.
expected_verdictagainst the signed vector's,which is authoritative.
vector_idvalues that are not in the signed catalogue.N/Arequires a non-emptyreason;PASSandFAILrequire anon-empty
observedobject.results/a2a-sepasses unchanged. To confirm the checks bite, a scratchsubmission with every verdict downgraded to
REVIEWand"observed": nullfails on the verdict cross-check, where previously itwould not have been examined at all.
Verified locally:
pytest tests/is 15 passed.This does not address self-attestation itself, which is a governance
question rather than a test one, but it does mean a submission has to be
wrong on purpose rather than by accident.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com