Make the corpus replay's own rules fail, and prove each one bites - #265
Merged
Conversation
The replay landed as four assertions over one fixed directory. Every one of them read the corpus this tree carries, that corpus is healthy, and a rule that has only ever run against a healthy subject cannot be told apart from a rule that refuses nothing. Two of #86's four conditions are statements about a run going RED - that an empty corpus reddens the build, and that a deliberately unhandled input in a seed reddens it - and neither of them had ever been watched happening. The rules now answer instead of asserting. `rules(root, named)` takes the corpus root and the target names as arguments and returns the refusals, so the gate hands it this tree's corpus and requires an empty answer while each proof hands it a root built for one defect and requires exactly that rule and no other. `replay_every_seed(root, call)` takes the target function the same way, so a proof can hand it a target that does not name what it did with a seed and watch the failure come out rather than be counted as an answer. The rules the gate runs and the rules the proofs trip are one function rather than two, which is what stops a proof passing against a second copy of the logic. What this prevents is a replay that reports a pass for a surface nobody reached. Every one of the four states - a root with no directory, a directory with no seed, a directory with no entry point behind it, and a target with no directory - exits zero and prints a page indistinguishable from a run that covered everything, and the same is true of a loop that catches a panic and counts it. Six deliberate removals were made against these rules and each reddened exactly one proof and left the other ten green. Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The issue this belongs to
#86, and it is not closed by this change. The reason is already written on the issue rather
than restated here: the first item of its produce list names the response
decoding as a target, there is no entry point for one in this tree, and a corpus
directory for it would be the empty-corpus state the same issue asks the build to
redden on.
What changed
The corpus replay landed as four assertions over one fixed directory. Each of
them read
tests/fixtures/corpus/, that corpus is healthy, and a rule that hasonly ever run against a healthy subject cannot be told apart from a rule that
refuses nothing. Two of #86's four conditions are statements about a run going
RED - that an empty corpus reddens the build, and that a deliberately unhandled
input in a seed reddens it - and neither had ever been watched happening.
The rules now answer instead of asserting.
rules(root, named)takes the corpusroot and the target names as arguments and returns the refusals it found, each as
a rule identifier and the subject it names. The gate hands it this tree's corpus
and the names in
TARGETSand requires the answer to be empty. Each proof handsit a root built for exactly one defect and requires the answer to name exactly
that rule and no other.
replay_every_seed(root, call)takes the target function the same way, so a proofcan hand it a target that does not name what it did with a seed and watch the
failure come out of the loop rather than be counted as an answer.
The rules the gate runs and the rules the proofs trip are one function rather than
two. A proof that ran against a second copy of the logic would prove the copy.
Three of the four rule assertions are now one test whose message names which
rules broke and on which subjects, so nothing that was asserted before is
asserted less: what moved is where the rule lives, not what it refuses.
What failure it prevents
A replay that reports a pass for a surface nobody reached. Four states do that
and all four exit zero: a root holding no directory, a directory holding no seed,
a directory with no entry point behind it, and a target with no directory. A
fifth is the loop itself - catching a panic and counting it turns an input the
code cannot name into a line in a report.
This is a guard hardened rather than a defect repaired. No such state has been
observed in this tree; the corpus was healthy before this change and is healthy
after it. What was missing was any run in which a rule here has ever refused
anything.
Evidence
The suite, at the commit being pushed:
Five of those eleven are the rules and the replay over this tree's corpus; six
are the proofs. The count in that binary before and after:
The other legs of the gate that read this file, run here:
One path this change touches:
What a guard here refuses, and the proof it bites
Six deliberate removals, one at a time, each reverted before the next. Every run
is
cargo test --locked --test replay_the_seed_corpusagainst the working treecarrying that one edit. These are runs made by hand rather than something a
reader re-derives from the tree: repeating one means making the same edit.
a-directory-holds-no-seeda_directory_with_no_seed_is_refusedthe-root-holds-no-directorya_root_with_no_directory_is_refuseda-directory-names-no-targeta_directory_naming_no_target_is_refuseda-target-has-no-directorya_target_with_no_directory_is_refusedlet answer = call(...)replaced by the same call undercatch_unwind, falling back to a stringa_seed_the_target_does_not_handle_reddens_the_replay!answer.is_empty()assertion insidereplay_every_seeda_target_that_answers_nothing_reddens_the_replayEach removal reddened exactly one proof and left the other ten green, which is
what says a proof is about its own rule rather than about the file.
Two of them were also run against the real corpus rather than against a root
built for the purpose, because #86's conditions are about this tree's corpus.
Both edits were reverted and
git statusis clean at the commit pushed.Emptying one real corpus directory, which is #86's
an empty corpus reddens the build:Adding a corpus directory with no entry point behind it, which is the state a
replay walks past while exiting zero:
The near-miss beside the panic proof is a test rather than a note.
the_same_root_without_that_seed_replays_greenhands the same target the sameroot with the one unhandled seed removed and requires a green replay of one seed,
so the red run beside it is about the seed rather than about a target that fails
at everything.
the_healthy_root_breaks_no_ruleis the same idea for the fourrules: the root each of them is a one-change neighbour of is asserted to break no
rule first, so nothing those proofs report can be attributed to their own base.
What this does not cover
#86 stays open after this. Its first produce item names the response decoding as
one of three fuzz targets and there is no entry point for one in this tree.
No fuzz target and no coverage-guided run. #86 asks for both, and neither is
in this change or in this tree. What is here is the replay half, which is the
part that gates, and it is now proven rather than only asserted.
The unhandled input is proven against a target built for the proof, not against
a parser in this crate. No parser here can be made to panic without changing
it, so the proof supplies a target that fails on one literal byte string. What
that proves is that the replay does not swallow such a failure. It says nothing
about whether any parser in
src/can be driven to one.A target nobody thought of is still not caught, and cannot be. A surface that
was never named has no corpus directory, so the empty-directory rule has nothing
to fail on. Two such surfaces are named on #86 already and the module
documentation names them where a reader of the file meets them.
Nothing measured performance. The replay binary reports 0.02 seconds and that
is what the harness printed, not a measurement anybody made a claim from.
Who has read it
Nobody but the author. There is no second reader on this board tonight, and the
runs above stand in place of one: six removals each reddening exactly one proof,
two demonstrations against the real corpus, and every gate leg that reads this
file run locally with its verdict pasted.
A correction to this body, made after the merge. The sentence that stood at
the second heading above read
It does not close #86.The linked-issue parsermatches the verb and the number and does not read the negation in front of them,
so merging this closed #86 as completed with two of its produce items unbuilt.
The issue is reopened and carries the reading. Both sentences here are rewritten
so the number never follows a closing verb, which is what the two landings before
this one did by writing
Refsand by leaving the number off the sentence.