chore(fuzz): add cargo-fuzz suite and untrusted-input coverage gate - #279
chore(fuzz): add cargo-fuzz suite and untrusted-input coverage gate#279crowecawcaw wants to merge 2 commits into
Conversation
bc3fef2 to
7113bad
Compare
| @@ -0,0 +1 @@ | |||
| -x No newline at end of file | |||
There was a problem hiding this comment.
Is there a way to do this that puts these into one file instead of hundreds of tiny files?
There was a problem hiding this comment.
The fuzzer needs separate files. We could combine them in a txt file and the split them at runtime, but its another step. I thinned out the starting seed cases instead.
Adds a coverage-guided libFuzzer suite (via cargo-fuzz) over the crates that parse, decode, or evaluate untrusted input, a CI workflow that runs it on PRs and merges to main, and a hard "all untrusted inputs are fuzzed" gate. Motivation: the quality-evaluation reports kept re-discovering the same class of defect by hand — arithmetic overflow and char-boundary panics reachable from attacker-controlled expression text, templates, and manifests. Encoding that class as executable checks stops it from being re-litigated every review cycle. Twelve fuzz targets, each asserting "any input returns Ok or Err — never panic, abort, or hang": expr_parse, expr_evaluate (both POSIX and Windows path formats), range_expr, int_range_new, range_expr_slice, expr_type_parse, format_string, copy_symbol_value, model_decode, model_create_job, snapshot_decode, snapshot_ops. The fuzz crate is outside the root workspace (its own empty [workspace] table) so the stable build/test/clippy/MSRV jobs are untouched; it builds only under a pinned nightly with AddressSanitizer, overflow-checks, and debug-assertions on. Curated seed corpora (mined from the crates' own tests, sample templates, and i64/multibyte edge cases) live in fuzz/seeds/ and are committed. scripts/run_fuzz.sh derives the target list from `cargo fuzz list`, so new targets are picked up by CI automatically with nothing to maintain. Fuzz-coverage gate (scripts/check_fuzz_coverage.py + fuzz/fuzz_coverage.toml, runs on stable in the Compliance CI job — no nightly, no fuzz build): every public function taking a &str / &[u8] / serde_json::Value / &Path parameter must be classified in the registry as either fuzzed-by-a-target or explicitly not-untrusted-with-a-reason. The check enforces no orphan/phantom targets, no dangling classifications, and — the ratchet — that a newly-added untrusted-input entry point fails CI until triaged. 108 input-shaped public functions are classified today (69 fuzzed, 39 not-untrusted). Also fixes one bug the suite found, with a regression test: path_starts_with under the Windows path format sliced a path by the base string's byte length, panicking when that offset fell inside a multibyte char (expr report finding X8), surfaced once expr_evaluate exercised the Windows path format. Fixed with a byte-slice comparison (equivalent for ASCII case-folding, boundary-safe). Note: earlier revisions of this branch also fixed RangeExpr/IntRange integer overflows the fuzzer found; those are now superseded by upstream OpenJobDescription#276 (bounded range values), so this branch rebases onto that and drops the redundant range changes, keeping the fuzz targets that guard the area. All openjd-expr tests pass; cargo clippy -D warnings is clean workspace-wide. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
| lib = crate_src / "lib.rs" | ||
| text = lib.read_text(encoding="utf-8", errors="replace") | ||
| private_tops = set() | ||
| for m in re.finditer(r"^\s*(pub\s+)?mod\s+([a-z_][a-z0-9_]*)\s*;", text, re.M): |
There was a problem hiding this comment.
The private-module regex only recognizes bare mod foo;; it misses restricted-visibility declarations like pub(crate) mod / pub(super) mod / pub(in path) mod. For pub(crate) mod budgeted_vec;, the (pub\s+)? group fails to match (there is no whitespace after pub), so the group matches empty and then mod fails against pub — the module is not added to private_tops.
Consequence: pub fns inside crate-private modules are scanned as "input-shaped public functions" even though they are not part of the crate’s public API. This is exactly what happens with crates/openjd-expr/src/edit_distance.rs::edit_distance / suggest_closest (a pub(crate) mod), which the docstring here says "must not count toward the untrusted surface" yet are forced into [[fuzzed]] at fuzz_coverage.toml:197-201.
The direction is safe (over-inclusion never lets a real entry point escape the ratchet), but it contradicts the documented contract and inflates the classified-surface count with crate-internal functions. Consider matching pub\s*(\([^)]*\))?\s+mod and treating any non-pub (i.e. anything with a (...) visibility restriction) as private.
… AGENTS.md and specs Per AGENTS.md report-driven development, strike through the report items this PR's cargo-fuzz suite resolves: - expr report: recommendation 20 (fuzz harness), the §4 structural gap, the §5 test_fuzz.py parity gap, and the X8 path_starts_with char-boundary panic (item 16 / recommendation 5), fixed in this PR with a regression test. - model report: gap 5 / recommendation 12 (fuzz testing for parser robustness), covered by the model_decode and model_create_job targets. Also document the new surface where developers will look for it: - AGENTS.md: quick-reference entries for scripts/run_fuzz.sh and scripts/check_fuzz_coverage.py, the Fuzz workflow and the Compliance job's fuzz-coverage gate in the CI table, and a note that new untrusted-input public fns must be classified in fuzz/fuzz_coverage.toml. - specs/architecture.md: fuzz/ in the project layout and a Key Design Decisions entry describing the suite and the coverage gate. The sessions report's directive-parsing fuzz note stays open — the suite does not cover openjd-sessions. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Summary
Adds a coverage-guided libFuzzer suite (via
cargo-fuzz) over the crates that parse, decode, or evaluate untrusted input, a CI workflow that runs it on PRs and merges tomain, and a hard "all untrusted inputs are fuzzed" gate.Why: the
reports/*-quality-evaluation-report.mdcycle keeps re-discovering the same class of defect by hand — arithmetic overflow and char-boundary panics reachable from attacker-controlled expression text, templates, and manifests. Encoding that class as executable checks stops it from being re-litigated every review cycle.Fuzz targets (12)
Each asserts the same invariant: any input returns
OkorErr— never panics, aborts, or hangs.expr_parseParsedExpression::new/with_profileexpr_evaluateevaluateunder both POSIX and Windows path formatsrange_exprRangeExpr::from_str+len/get/iterint_range_newIntRange::newwith arbitraryi64triplesrange_expr_sliceRangeExpr::slicewith arbitraryi64indicesexpr_type_parseExprType::parseformat_stringFormatString::new+resolve_string_withcopy_symbol_valuecopy_symbol_valuedotted-name walkmodel_decodedocument_string_to_object+decode_{job,environment,}_templatemodel_create_jobpreprocess_job_parameters+create_job(instantiation — one layer past decode)snapshot_decodedecode_manifest+Manifest::validatesnapshot_opsdiff/compose/partition/subtree/filterover decoded manifestsThe fuzz crate lives outside the root workspace (its own empty
[workspace]table), so the stable build/test/clippy/MSRV jobs are untouched. It builds only under a pinned nightly with AddressSanitizer,overflow-checks, anddebug-assertionson.Coverage gate (the hard check)
scripts/check_fuzz_coverage.py+fuzz/fuzz_coverage.toml, wired into the Compliance CI job — pure source+manifest analysis on stable Python, no Rust build, no nightly. Every public function taking a&str/&[u8]/serde_json::Value/&Pathparameter must be classified in the registry as either fuzzed-by-a-target or explicitly not-untrusted-with-a-reason. The check enforces:fuzz_targets/*.rsmissing from the registry),108 input-shaped public functions are classified today (69 fuzzed, 39 not-untrusted). All four failure modes were verified to fail CI.
CI / running
FUZZ_SECONDS=10per target) on PRs and merges tomain.scripts/run_fuzz.shderives the target list fromcargo fuzz list, so new targets are picked up automatically — nothing to maintain in YAML.scripts/run_fuzz.sh(all targets),FUZZ_SECONDS=300 scripts/run_fuzz.sh(deeper), or name specific targets. Seefuzz/README.md.Seeds
Curated, per-file, one per structurally-distinct input shape plus deliberate edge cases (multibyte strings,
i64boundaries, symbol-table-prefixed expressions) — 189 files total, in the typical range for committed fuzz seeds..gitignoreexcludes libFuzzer's SHA-named discovered inputs so they can't leak into the tree.Bug found & fixed
path_starts_withunder the Windows path format sliced a path by the base string's byte length, panicking when that offset fell inside a multibyte char (expr report finding X8) — surfaced onceexpr_evaluateexercised the Windows path format. Fixed with a byte-slice comparison (equivalent for ASCII case-folding, boundary-safe), plus a regression test.Rebase note
Earlier revisions of this branch also fixed
RangeExpr/IntRangeinteger overflows the fuzzer found. Those are now superseded by upstream #276 (bounded range values), so this branch is rebased onto it and drops the redundant range changes — keeping the fuzz targets (int_range_new,range_expr_slice) that guard the area.All
openjd-exprtests pass;cargo clippy -D warningsis clean workspace-wide.🤖 Draft — opened for review, not yet marked ready.