Add advanced fuzz targets to verify more invariants (round-trip, framing, state-machine…)

[CBenoit]

Move beyond “bytes-only decode fuzzing” to higher-signal fuzzing that asserts invariants and exercises state machines.

Proposed fuzz targets

1. round_trip (typed):

• Generate a structured PDU (Arbitrary)
• Encode -> Decode -> Compare (plus size invariants)

2. message_decoding_invariants (bytes):

• Feed raw bytes into decoder(s)
• If decoding succeeds, assert framing/size/consumption invariants

3. connector_state_machine (optional in this issue; can be a follow-up):

• Generate sequences / frames and step connector state machines
• Validate buffer bookkeeping and hint-based framing invariants

Tasks

• Add round_trip fuzz target wired to an oracle module
• Add at least one non-trivial oracle invariant (not just “no panic”)
• Ensure targets can be run via cargo xtask fuzz run <target>
• Add regression test mechanism: replay minimized crashers in unit tests

“Round-trip” oracle (typed fuzzing)

This is a high signal harness: generate a structured PDU, encode it, decode it back, compare.

Fuzz target (tiny):

#![no_main]
use libfuzzer_sys::fuzz_target;

use ironrdp_fuzzing::{generators::AnyPdu, oracles};

fuzz_target!(|msg: AnyPdu<'_>| {
    oracles::round_trip(msg);
});

Generator idea (AnyPdu<'a>):

• an enum that contains a curated set of PDUs we want to round-trip (start small; expand over time).

• it should provide:

  • encode() (to Vec<u8>)
  • decode_owned(bytes) (back into an owned representation)
  • into_owned() for comparison

Oracle idea:

• encode
• decode_owned
• assert_eq!(decoded, original.into_owned())
• also assert size invariants (encoded.len() == original.size()) where applicable

This is the harness that typically finds:

• wrong length computations
• missing fields in encode/decode
• decode accepting invalid encodings (or encode producing something decode can’t parse)

“Message decoding” oracle (raw bytes, but with invariants)

We keep a raw-bytes decoder too, but it should assert invariants, e.g.:

• if decoding succeeds, then “frame-size detection” should succeed and be consistent
• if decode consumes N bytes, that should match computed/expected sizes
• avoid silent under/over-reads

This is especially valuable for:

• frame boundary logic
• “find size” / “hint” logic
• partial frame handling

State-machine fuzzing (connector sequences)

We also want fuzzing that targets state machines (connector / acceptor / channel sequences), not just stateless PDUs.

The typical pattern:

• generate a connector + an input payload

• capture the connector’s “expected next PDU hint”

• call step(payload, out_buf)

• if it succeeds:

  • verify output length bookkeeping is correct (buffer length matches returned “written length”)
  • verify that the hint can indeed determine a frame length for payload (since the step succeeded)

We can start with “single-step” fuzzing and later move to “multi-step” inputs (a sequence of frames) to reach deeper states.

Acceptance criteria

• Oracles assert invariants (round-trip equality and/or framing invariants)
• Running the fuzz target locally is documented

[glamberson]

Task 2 (message_decoding_invariants) is shipped via #1320. Asserts pdu.size() == encode_vec(&pdu).len() on every decoder-accepted input — the Encode::size() soundness contract; libFuzzer treats violations as crashes per the oracle module doc.

Task 1 (round_trip) was shipped via #1291 (covers ironrdp-pdu types) plus #1317 (extends the round-trip pattern to ironrdp-egfx types under the egfx fuzz-coverage child umbrella at #1316).

Task 3 (connector_state_machine) — marked "optional in this issue; can be a follow-up" in the body — remains open. With tasks 1 and 2 shipped, this issue can close if task 3 is moving to its own scope, or stays open if task 3 is in-scope here. Suggesting closure; happy to drive task 3 as a separate sibling issue under #1120 if you want it tracked explicitly with its own scope discussion.

[CBenoit]

Greg Lamberson (@glamberson) I’m fine with keeping the current issue, or opening a new one! Did you perform an analysis of the benefits of connector_state_machine? Do you think there is something nice to do for the state machines?