Skip to content

Harden the decode path against malformed input + add a fuzzer - #55

Merged
treeform merged 5 commits into
masterfrom
net-hard
Jul 14, 2026
Merged

Harden the decode path against malformed input + add a fuzzer#55
treeform merged 5 commits into
masterfrom
net-hard

Conversation

@treeform

@treeform treeform commented Jul 14, 2026

Copy link
Copy Markdown
Owner

What

Makes fromFlatty safe to run on untrusted/hostile bytes (e.g. packets off the wire). Today a malformed blob can take the whole process down; after this PR a bad blob fails with a catchable error instead of a segfault, an out-of-memory abort, or a stack overflow. Adds a fuzzer that proves it.

Why

fromFlatty is commonly used to decode network data, but the decode path trusted its input: unchecked reads, unchecked length/count prefixes driving allocations, cast-ed enum discriminators, and unbounded recursion. A fuzzer (see below) found 178 hard aborts across every length-prefixed and variant type — each one a remotely triggerable crash.

The fuzzer (tests/fuzz.nim)

Feeds truncated, oversized, and randomly-mutated blobs to the decode path across every type shape and asserts none of them crash the process (a catchable error is fine). Because a bad input can hard-abort, each child decode runs in a subprocess that checkpoints its input first — so an uncatchable abort is reported with a replayable reproducer instead of killing the run. Deterministic (seeded), so every finding replays.

nim r tests/fuzz.nim                         # 0 hard aborts after this PR
nim r tests/fuzz.nim --replay <type> <hex>   # replay one reproducer

The hardening

  • Bounds-check every read (binny): truncated blobs can't over-read; stays on even under -d:danger.
  • Validate length/count prefixes before allocating (string, seq, Table, Set): a bogus/negative/huge length can't drive a giant allocation or over-read.
  • Range-check enum discriminators before building an object variant: an out-of-range tag can no longer corrupt a variant or segfault via new(x, badTag).
  • Cap Table/HashSet preallocation from an untrusted count: fixes a measured 55–69× memory-amplification DoS (a ~0.5 MB blob forced ~33 MB; ~5 MB forced ~270 MB). Preallocation is now a bounded constant; large legitimate containers still grow as their real entries decode.
  • Stack-pointer watermark stops deeply nested input (a long ref/seq chain) before it overflows the thread stack — no per-call depth counter, no bookkeeping to unwind. Guards only the four heap-indirection procs (ref/seq/Table/Set), which every type cycle must pass through, so object/tuple/array stay untouched.

Decode failures raise FlattyError (bad length/count/enum) or IndexDefect (truncation); catch both.

Performance

tests/bench_flatty_types.nim, -d:release, best-of-5. Serialize is unchanged (no checks added there). Deserialize:

  • Bulk binary paths (array/seq of PODs, strings): no measurable cost — one check per blob, then the same copyMem.
  • Structured/pointer-heavy paths pay for the per-field bounds check: ~1.15–1.2× for seq-of-struct / Table / seq[string], up to ~1.4–1.5× for the ref-object tree. Absolute cost is tens of microseconds on 100–380 KB payloads — it's the untrusted-input path.

The stack guard and the Table prealloc cap add no measurable cost on normal data.

@treeform
treeform merged commit 5c815ec into master Jul 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant