Replay a seed corpus over the three parsers that have an entry point - #258
Merged
Conversation
Two questions get confused with each other and this answers one. A coverage-guided run looks for new inputs and is slow. Replaying the seeds already known to be hostile is fast, and it catches the case where a change makes a known input do something the code does not name. Only the second is here, it runs inside `cargo test --locked`, and the `test` check already carries it. The target list is derived from the corpus directories rather than written in the harness, which is what #86 asks for so a target added by acquiring a corpus is not silently uncovered by a list nobody updated. Both directions are refused: a directory with no entry point behind it is a corpus nobody replays, and a target with an entry point and no directory is a target nothing replays. An empty directory and an empty root are refused too, because replaying nothing proves nothing and exits zero, which reads exactly like a run that found no defect. Three targets, and they are the three surfaces that take bytes today. The image format handling 0055 puts in front of a decoder, the envelope 0105 puts on every byte read back out of a client-supplied store, and the address parser. The first two are on 0101's untrusted list by name; the third is not, and the reason it is here is robustness rather than the threat model, which is written where the target is so a directory name does not have to carry it. Twenty-six seeds, named for the answer each was built to reach. A seed asserts that the target ANSWERS, because each of the three returns a value or a member of a closed refusal set, so anything the code does not name could only arrive as a panic. A separate assertion reads each closed set out of the crate and requires the corpus as a whole to reach every member, which is what stops a corpus decaying into a list of inputs that all fail the same way. Four deliberate violations, each reddening what it should: mkdir tests/fixtures/corpus/response-decoding the corpus directory response-decoding holds no seed. #86 asks that an empty corpus redden the build rather than passing quietly assert!(!bytes.is_empty(), "an unnamed exception on a seed"); // inside admitted() panicked at src\artwork\format.rs:280:5 test every_seed_is_replayed_and_every_target_answers ... FAILED TARGETS with "response-decoding" added and no directory made targets with no corpus directory: ["response-decoding"] one seed removed no seed under cache-envelope reaches Digest. The seeds that are there reach {Malformed, Version, Kind, Length}. Refs #86. Its own produce list names the response decoding as a target and there is no decoder to point at, and the coverage-guided half needs a fuzzing runtime nobody has argued for under 0103, so this closes nothing. Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The issue this belongs to
Refs #86. It does not close it, and the reason is under "What this does not
cover" below.
What changed
tests/fixtures/corpus/holds a seed corpus andtests/replay_the_seed_corpus.rsreplays it insidecargo test --locked, whichis what the gating
testcheck already runs.#86 separates two questions that get confused with each other, and only one of
them is here. A coverage-guided run looks for NEW inputs and is slow, which is why
the gate this board is measured against schedules it and does not gate on it.
Replaying the seeds already known to be hostile is fast, and it catches the case
where a change makes a known input do something the code does not name. This is
the second.
The target list is derived from the corpus directories rather than written in
the harness, which is the shape #86 asks for so that a target added by acquiring a
corpus cannot be silently uncovered by a list nobody updated. Both directions are
refused: a directory with no entry point behind it is a corpus nobody can replay,
and a target with an entry point and no directory is a target nothing replays. An
empty directory and an empty root are refused for the same reason - replaying
nothing proves nothing and exits zero, which reads exactly like a run that found
no defect.
Three targets, which are the three surfaces that take bytes today. The image
format handling 0055
puts in front of a decoder, the envelope
0105 puts on every
byte read back out of a client-supplied store, and the address parser. The first
two are on
0101's untrusted list by name. The
third is not, and the earlier reading on #86 is right about that: 0101 trusts a
client for passing on what the operator typed. The reason it is a target anyway is
robustness rather than the threat model - that no string makes
parsedosomething other than answer - and that reason is written at the target, because
#86's own derivation means a directory name carries no reason and whoever adds a
target picks the reason by picking it.
What a seed asserts is that the target ANSWERS. Each of the three returns a
value or a member of a closed refusal set, so an input reaching something the code
does not name could only arrive as a panic, and a panic in a replay is this test
failing. That is #86's
failing on any unnamed exceptionin the terms thislanguage offers.
A second assertion reads each closed set out of the crate -
WhichCheckFailed::all()and
Refused's members - and requires the corpus AS A WHOLE to reach every one ofthem, plus at least one input that is accepted. That is what stops a corpus
decaying into a list of inputs that all fail the same way, which is where a corpus
ends up when seeds are added without anybody asking what each is for. Individual
seeds are deliberately not pinned to an outcome, because that would put a second
copy of each module's table in this file.
Evidence
At
7abb27b, twenty-six seeds across three targets:and the suite, which now carries a fifth target:
The analyser, the formatter, and the three shell legs that run on this machine
and touch these paths:
The seeds go in under
tests/fixtures/, so.gitattributesalready covers theirbytes with
-textand the rule was written for exactly this: its own comment saysa fixture added in a subdirectory tomorrow is covered without that file being
edited.
The guards refusing
Four deliberate violations, each reddening what it names.
An empty corpus directory:
A seed reaching an answer the code does not name, which is what the replay is
for. A panic put inside
admittedfor one input:A target with an entry point and no corpus, which is the direction a
corpus-derived list does not catch on its own:
A named refusal no seed reaches any more, with one seed removed:
The tree is restored in each case, and the runs above are the restored tree.
The means
Rust, in the test crate the gate already runs, and files on disk for the seeds.
A separate fuzzing harness was NOT the means and that is the load-bearing half of
this choice: a coverage-guided runtime is a dependency, 0103 admits one only with
the clause that admitted it and the condition that retires it written beside the
entry, and nothing on this board has argued one for this purpose. What #86 asks to
run inside the GATING build is a replay, a replay is a loop over files, and it
needs nothing the tree does not already carry. The crate graph is untouched.
What this does NOT cover
#86 is not closed by this. Its produce list names the response decoding as one
of the three targets and there is no decoder to point at, so a corpus directory
for it would be the empty-corpus state the third bullet asks the build to redden
on rather than a target. Its condition also asks for a coverage-guided run, which
needs the runtime named above.
A target nobody thought of is not caught and cannot be. Deriving the list from
the directories protects against a list nobody updated; it does not protect
against a surface that was never named, because such a surface has no directory
and the empty-directory rule has nothing to fail on. Two are already named on #86 -
0116's change-notification connection and the response decoding - and the module
documentation names both so a reader is not left to infer the bound.
No seed reaches
TheEncodedLengthPassedItsBound. Reaching it means committingsixteen mebibytes, and 0055 applies that bound during a transfer rather than to a
file on disk. The closed-set assertion skips that one member by name with the
reason on the line, so the exclusion is read beside the verdict rather than found
later.
The corpus holds no input that has ever caused a defect here, because none has
been recorded. #86 asks for every such input to be a seed; what is here is one
seed per named answer, which is a different and weaker basis, and a real defect
found later still owes its seed.
Nothing asserts WHICH answer a seed produces. A change moving a seed from
LengthtoDigestis caught only if it empties one of the sets, and the reasonfor that trade is written in the file.
No second reader. Nobody other than the author has read this change.