Add an ALLOW verdict so the corpus can carry negative controls - #4
Open
laxsharma wants to merge 2 commits into
Open
Add an ALLOW verdict so the corpus can carry negative controls#4laxsharma wants to merge 2 commits into
laxsharma wants to merge 2 commits into
Conversation
All five v0 vectors are `BLOCK` or `REVIEW`, and `docs/vector-format.md` makes both satisfiable by an HTTP 4xx. A rail that returns 400 to every request therefore scores 100 percent, and the corpus has no way to say so. That matters most for the gates these vectors actually test. For dispute rate limiting and reputation velocity, the false positive is the real product risk: an over-eager dispute gate is itself the liquidity-freeze denial of service that motivated the dispute-DoS category in the first place. Right now the corpus measures one side of the confusion matrix. This adds the verdict and documents the pairing, without adding vectors: - `validate_vector_shape` accepts `ALLOW`. - README verdict table gains the row plus a note on why. - `docs/vector-format.md` gains an `ALLOW` pass condition and a "Negative controls" section listing a concrete counterpart for each of the five current vectors, for example a legitimate second partial release within the remaining balance to pair with `escrow_double_release`. No behaviour changes for existing vectors and no existing vector is touched, so this is inert until someone lands `ALLOW` fixtures. I would be glad to write those five fixtures, but I cannot sign them: `harness/jws.py` pins one key, so any vector not signed by `did:web:api.algovoi.co.uk` fails CI, and `docs/vector-format.md` already notes contributors must "coordinate with maintainers for signing". If there is appetite for a signer registry in `manifest.json` (a list rather than a single `kid`, with the harness resolving `signer_did` per vector), that would unblock this and any other outside contribution. Happy to send that as a separate PR if it is wanted, or to hand the fixtures to @chopmob-cloud for signing under the existing key, whichever fits better. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an ALLOW verdict so the corpus can carry negative controls
Every v0 vector expects
BLOCKorREVIEW, and both are satisfied by anHTTP 4xx. A rail that rejected every request would therefore score 100
percent on the current corpus while being useless in production, and
nothing in the format lets a submission demonstrate otherwise.
For the gates these vectors exercise, dispute rate limiting, reputation
velocity, refund authorization, the false-positive rate is the dominant
product risk. It is the one property the corpus cannot currently measure.
Changes:
expected_verdictacceptsALLOWin the shape validator.ALLOWpasses when the legitimate operation completes;blocking it, or holding it for review, is
FAIL.docs/vector-format.mdwith acandidate
ALLOWcase paired to each of the five existing vectors, forexample a genuine second partial release on an escrow in
partially_releasedas the counterpart toescrow_double_release.lowercase and near-miss strings.
No vectors are added here, and no existing vector or result changes.
Signing is the blocker for contributing the vectors themselves:
harness/jws.pypins a single key, so anything not signed bydid:web:api.algovoi.co.ukfails CI. Happy to draft theALLOWfixturesfor signing if that is the preferred order, or to work on multi-signer
support first if you would rather open contribution up.
Verified locally:
pytest tests/is 22 passed, andpython -m harness.validate_vectors --vectors vectors/v0reports all 5valid.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com