Malformed handshake extensions poison the transcript: validate full message before consuming/transcript-append

[lolgesten]

Summary

The recent parser-hardening commits (ff2dc1c, c2cda09, 2cfc96f, 831efad, and related) added strict rejection of malformed extensions, but the rejection happens after the enclosing handshake message has already been consumed — marked handled, peer sequence number advanced, and appended to the transcript. When the inner extension parse then fails, the error is (correctly) swallowed as transient, but the damage is already done: the corrupt message is in the transcript and the sequence has advanced.

Swallowing the error is the right behavior — over UDP, malformed/forged incoming data is expected and must not tear down the DTLS session. The actual defect is upstream: corrupt data must not be added to the transcript (or advance handshake state) in the first place. The full message body, including its inner extensions, must be validated before it is consumed.

Mechanism

1. A handshake message arrives. next_handshake / defragment marks it handled, advances peer_handshake_seq_no (src/dtls12/engine.rs:664), and appends the raw bytes to the transcript (src/dtls13/message/handshake.rs:169).
2. Only then does the handler parse the inner extensions (src/dtls12/client.rs:514, src/dtls12/server.rs:427,436,442; src/dtls13/server.rs:453-496).
3. A malformed inner extension produces a nom::Err, classified as Transient (src/error.rs:704) and swallowed as Ok(()) (src/error.rs:687-693). This swallowing is correct.
4. But the message has already polluted the transcript and advanced state.

Impact

During the plaintext (unauthenticated) early handshake, a forged packet carrying a structurally-malformed extension (e.g. a truncated use_srtp / supported_groups / key_share) is consumed before validation:

• DTLS 1.2 (client/server): peer_handshake_seq_no advances, then the parse fails and is swallowed. The genuine retransmission with the same message_seq is later dropped as an old duplicate (src/dtls12/engine.rs:282). Handshake stalls until the connect timeout fires.
• DTLS 1.3 (server await_client_hello): the ClientHello is appended to the transcript before the inner parse fails. A later legitimate ClientHello replaces the handled entry but is appended to the now-poisoned transcript, so the handshake ultimately fails at Finished verification.

Net effect is a handshake-stall DoS. On-path this is trivial; off-path it requires winning the injection race against the genuine message (the standard DTLS plaintext-injection caveat).

Recommendation

Do not consume / advance handshake state or append to the transcript until the full message body — including all inner extensions — has parsed successfully.

Validate the complete message body first; only on success mark it handled, advance peer_handshake_seq_no, and append to the transcript. If any part fails to parse, drop the packet (swallow as transient, as today) leaving handshake state untouched, so a subsequent genuine retransmission of the same message_seq is processed normally.

This keeps the correct UDP behavior (malformed/forged data is silently discarded, never tears down the session) while ensuring corrupt data can neither poison the transcript nor desynchronize sequence tracking.

Affected locations

• Consume-before-validate (DTLS 1.2): src/dtls12/engine.rs:664, with duplicate-drop at src/dtls12/engine.rs:282; inner parses at src/dtls12/client.rs:514, src/dtls12/server.rs:427,436,442
• Consume-before-validate (DTLS 1.3): src/dtls13/message/handshake.rs:169; inner parses at src/dtls13/server.rs:453-496
• Transient swallow (correct, keep as-is): src/error.rs:687-693, src/error.rs:704-712

Notes

Severity is bounded: this only affects the early (pre-encryption) handshake window. The fix should not surface these as fatal errors — discarding malformed UDP input is correct; the change is purely to defer state mutation until after successful validation.

Found during a security review of the recent parser-hardening / error-restructure commit series.