Skip to content

Replay a seed corpus over the three parsers that have an entry point - #258

Merged
iderex merged 1 commit into
mainfrom
the-seed-corpus-and-its-replay-86
Aug 31, 2026
Merged

Replay a seed corpus over the three parsers that have an entry point#258
iderex merged 1 commit into
mainfrom
the-seed-corpus-and-its-replay-86

Conversation

@iderex

@iderex iderex commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

The issue this belongs to

Refs #86. It does not close it, and the reason is under "What this does not
cover" below.

What changed

tests/fixtures/corpus/ holds a seed corpus and
tests/replay_the_seed_corpus.rs replays it inside cargo test --locked, which
is what the gating test check already runs.

#86 separates two questions that get confused with each other, and only one of
them is here. A coverage-guided run looks for NEW inputs and is slow, which is why
the gate this board is measured against schedules it and does not gate on it.
Replaying the seeds already known to be hostile is fast, and it catches the case
where a change makes a known input do something the code does not name. This is
the second.

The target list is derived from the corpus directories rather than written in
the harness, which is the shape #86 asks for so that a target added by acquiring a
corpus cannot be silently uncovered by a list nobody updated. Both directions are
refused: a directory with no entry point behind it is a corpus nobody can replay,
and a target with an entry point and no directory is a target nothing replays. An
empty directory and an empty root are refused for the same reason - replaying
nothing proves nothing and exits zero, which reads exactly like a run that found
no defect.

Three targets, which are the three surfaces that take bytes today. The image
format handling 0055
puts in front of a decoder, the envelope
0105 puts on every
byte read back out of a client-supplied store, and the address parser. The first
two are on
0101's untrusted list by name. The
third is not, and the earlier reading on #86 is right about that: 0101 trusts a
client for passing on what the operator typed. The reason it is a target anyway is
robustness rather than the threat model - that no string makes parse do
something other than answer - and that reason is written at the target, because
#86's own derivation means a directory name carries no reason and whoever adds a
target picks the reason by picking it.

What a seed asserts is that the target ANSWERS. Each of the three returns a
value or a member of a closed refusal set, so an input reaching something the code
does not name could only arrive as a panic, and a panic in a replay is this test
failing. That is #86's failing on any unnamed exception in the terms this
language offers.

A second assertion reads each closed set out of the crate - WhichCheckFailed::all()
and Refused's members - and requires the corpus AS A WHOLE to reach every one of
them, plus at least one input that is accepted. That is what stops a corpus
decaying into a list of inputs that all fail the same way, which is where a corpus
ends up when seeds are added without anybody asking what each is for. Individual
seeds are deliberately not pinned to an outcome, because that would put a second
copy of each module's table in this file.

Evidence

At 7abb27b, twenty-six seeds across three targets:

for d in tests/fixtures/corpus/*/; do echo "$d $(ls "$d" | wc -l)"; done
tests/fixtures/corpus/artwork-format/ 8
tests/fixtures/corpus/cache-envelope/ 9
tests/fixtures/corpus/server-address/ 9

and the suite, which now carries a fifth target:

cargo test --locked | grep 'test result'
test result: ok. 429 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 28 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 97 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The analyser, the formatter, and the three shell legs that run on this machine
and touch these paths:

cargo clippy --locked --all-targets -- -D clippy::all -D clippy::pedantic -D clippy::cargo ; echo "exit=$?"
exit=0

cargo fmt --check ; echo "exit=$?"
exit=0

bash .github/fixture-scrub/fixture-scrub.sh check | tail -1
Every rule above was applied to its subject and refused nothing.

bash .github/invariants/invariants.sh check | tail -1
Every rule above was applied to its subject and refused nothing.

bash .github/doc-paths/doc-paths.sh check | tail -1
Every path these documents name resolves against the tracked set.

The seeds go in under tests/fixtures/, so .gitattributes already covers their
bytes with -text and the rule was written for exactly this: its own comment says
a fixture added in a subdirectory tomorrow is covered without that file being
edited.

The guards refusing

Four deliberate violations, each reddening what it names.

An empty corpus directory:

mkdir tests/fixtures/corpus/response-decoding

thread 'no_corpus_directory_is_empty' panicked:
the corpus directory response-decoding holds no seed. #86 asks that an empty
corpus redden the build rather than passing quietly, because replaying nothing
proves nothing.
test result: FAILED. 3 passed; 2 failed

A seed reaching an answer the code does not name, which is what the replay is
for. A panic put inside admitted for one input:

+    assert!(!bytes.is_empty(), "an unnamed exception on a seed");

thread 'every_seed_is_replayed_and_every_target_answers' panicked at src\artwork\format.rs:280:5:
an unnamed exception on a seed
test result: FAILED. 3 passed; 2 failed

A target with an entry point and no corpus, which is the direction a
corpus-derived list does not catch on its own:

-const TARGETS: &[&str] = &["artwork-format", "cache-envelope", "server-address"];
+const TARGETS: &[&str] = &["artwork-format", "cache-envelope", "server-address", "response-decoding"];

targets with no corpus directory: ["response-decoding"]. A target with no seeds is
a target nothing replays, and the run above would have passed without reaching it.
test result: FAILED. 4 passed; 1 failed

A named refusal no seed reaches any more, with one seed removed:

no seed under cache-envelope reaches Digest. The seeds that are there reach
{Malformed, Version, Kind, Length}.
test result: FAILED. 4 passed; 1 failed

The tree is restored in each case, and the runs above are the restored tree.

The means

Rust, in the test crate the gate already runs, and files on disk for the seeds.
A separate fuzzing harness was NOT the means and that is the load-bearing half of
this choice: a coverage-guided runtime is a dependency, 0103 admits one only with
the clause that admitted it and the condition that retires it written beside the
entry, and nothing on this board has argued one for this purpose. What #86 asks to
run inside the GATING build is a replay, a replay is a loop over files, and it
needs nothing the tree does not already carry. The crate graph is untouched.

What this does NOT cover

#86 is not closed by this. Its produce list names the response decoding as one
of the three targets and there is no decoder to point at, so a corpus directory
for it would be the empty-corpus state the third bullet asks the build to redden
on rather than a target. Its condition also asks for a coverage-guided run, which
needs the runtime named above.

A target nobody thought of is not caught and cannot be. Deriving the list from
the directories protects against a list nobody updated; it does not protect
against a surface that was never named, because such a surface has no directory
and the empty-directory rule has nothing to fail on. Two are already named on #86 -
0116's change-notification connection and the response decoding - and the module
documentation names both so a reader is not left to infer the bound.

No seed reaches TheEncodedLengthPassedItsBound. Reaching it means committing
sixteen mebibytes, and 0055 applies that bound during a transfer rather than to a
file on disk. The closed-set assertion skips that one member by name with the
reason on the line, so the exclusion is read beside the verdict rather than found
later.

The corpus holds no input that has ever caused a defect here, because none has
been recorded. #86 asks for every such input to be a seed; what is here is one
seed per named answer, which is a different and weaker basis, and a real defect
found later still owes its seed.

Nothing asserts WHICH answer a seed produces. A change moving a seed from
Length to Digest is caught only if it empties one of the sets, and the reason
for that trade is written in the file.

No second reader. Nobody other than the author has read this change.

Two questions get confused with each other and this answers one. A
coverage-guided run looks for new inputs and is slow. Replaying the seeds already
known to be hostile is fast, and it catches the case where a change makes a known
input do something the code does not name. Only the second is here, it runs
inside `cargo test --locked`, and the `test` check already carries it.

The target list is derived from the corpus directories rather than written in the
harness, which is what #86 asks for so a target added by acquiring a corpus is not
silently uncovered by a list nobody updated. Both directions are refused: a
directory with no entry point behind it is a corpus nobody replays, and a target
with an entry point and no directory is a target nothing replays. An empty
directory and an empty root are refused too, because replaying nothing proves
nothing and exits zero, which reads exactly like a run that found no defect.

Three targets, and they are the three surfaces that take bytes today. The image
format handling 0055 puts in front of a decoder, the envelope 0105 puts on every
byte read back out of a client-supplied store, and the address parser. The first
two are on 0101's untrusted list by name; the third is not, and the reason it is
here is robustness rather than the threat model, which is written where the target
is so a directory name does not have to carry it.

Twenty-six seeds, named for the answer each was built to reach. A seed asserts
that the target ANSWERS, because each of the three returns a value or a member of
a closed refusal set, so anything the code does not name could only arrive as a
panic. A separate assertion reads each closed set out of the crate and requires
the corpus as a whole to reach every member, which is what stops a corpus decaying
into a list of inputs that all fail the same way.

Four deliberate violations, each reddening what it should:

    mkdir tests/fixtures/corpus/response-decoding
    the corpus directory response-decoding holds no seed. #86 asks that an empty
    corpus redden the build rather than passing quietly

    assert!(!bytes.is_empty(), "an unnamed exception on a seed");   // inside admitted()
    panicked at src\artwork\format.rs:280:5
    test every_seed_is_replayed_and_every_target_answers ... FAILED

    TARGETS with "response-decoding" added and no directory made
    targets with no corpus directory: ["response-decoding"]

    one seed removed
    no seed under cache-envelope reaches Digest. The seeds that are there reach
    {Malformed, Version, Kind, Length}.

Refs #86. Its own produce list names the response decoding as a target and there
is no decoder to point at, and the coverage-guided half needs a fuzzing runtime
nobody has argued for under 0103, so this closes nothing.

Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
@iderex
iderex merged commit b93d599 into main Aug 31, 2026
24 checks passed
@iderex
iderex deleted the the-seed-corpus-and-its-replay-86 branch August 31, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant