Harden WebSocket framing and fragmentation handling

[mbuckton]

Summary

• replace the WebSocket endpoint's shared single-buffer framing logic with an incremental RFC 6455 decoder
• preserve frame state across arbitrary TCP segmentation, including split headers, extended lengths, mask keys and payloads
• retain fragmented messages until the final fragment arrives and the complete text message passes UTF-8 validation
• support continuation frames with interleaved control frames
• validate client masking, RSV bits, canonical lengths, control-frame constraints, close payloads and protocol close codes
• cap complete WebSocket messages at 64 MiB and fail oversized messages with close code 1009
• serialize outbound WebSocket headers and payloads into one pending frame so partial socket writes cannot introduce duplicate headers or stale buffer bytes
• return PONG only for PING and echo the PING payload
• preserve WebSocket bytes coalesced with the HTTP upgrade request
• make the HTTP upgrade parser incremental, bounded to 8 KiB and validate the RFC 6455 request fields
• route STOMP heartbeats through the normal selector write queue
• add a narrow buffered-read endpoint contract so validated messages larger than a protocol read buffer drain without waiting for another TCP readiness event
• remove the obsolete WebSocketHeader and WebSocketPacket implementations that contained the original unsafe logic

Root cause

The captured STOMP SEND arrived as an 8-byte WebSocket header followed by the 1,183-byte masked payload in the next TCP segment. The old endpoint cleared the parsed header when no payload bytes were present. It then parsed the first masked payload byte as a PONG opcode and sent an unflipped shared write buffer, producing stale bytes followed by zero-filled capacity. The browser closed immediately after receiving that malformed frame.

Tests added

Regression coverage has been added for:

• the exact header-only then 1,183-byte payload split seen in the capture
• one-byte-at-a-time network reads
• fragmented HTTP upgrades and upgrade/frame coalescing
• bounded upgrade-header size
• fragmented text withheld until FIN
• interleaved PING during fragmented messages
• UTF-8 validation across fragment boundaries
• invalid UTF-8 not being exposed to STOMP
• canonical extended payload lengths
• payload delivery across multiple protocol buffers without another socket read
• unmasked and invalid continuation/control frames
• protocol-error and normal close handshakes
• partial and initially blocked outbound socket writes
• one-byte STOMP heartbeat payloads under write backpressure
• valid PONG framing with no zero padding
• a raw port-8674 STOMP/WebSocket integration case reproducing the captured split

Validation

Per request, the Maven test suite and full server tests were not run because they pull and execute the complete server test environment. This PR has been reviewed statically and remains draft for validation in the project's normal targeted build/test workflow.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f3a9100f-f934-4c2c-9d22-e13b3c65b206

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}