Skip to content

[Test] No test asserts that a large message does not delay unrelated traffic on the same connection #1385

Description

@pathosDev

Use case

The cluster transport is one TCP connection per peer with a length-prefixed frame decoder. Frames are read and dispatched in arrival order, and there is no separate lane for large payloads — grep for chunking, multiplexing or lane separation across src/cluster/Transport.ts and src/cluster/Protocol.ts finds only TCP read chunks, which is a different thing.

So a single large message occupies the connection for as long as it takes to transfer, and everything behind it waits. That includes gossip and heartbeats.

The consequence is not a slow message; it is a false failure detection. Heartbeats delayed behind a multi-megabyte payload look exactly like a peer that has stopped responding, and the phi-accrual detector will eventually mark the peer unreachable. A downing strategy then acts on that. A large but entirely legitimate message can therefore take a healthy node out of the cluster.

Nothing tests this. There is a frame size cap, which bounds the worst case, and tests/unit/Protocol.test.ts asserts that decode cost stays linear in the bytes received — both good, and neither is the question. The question is whether unrelated traffic on the same connection is delayed, and it has never been asked.

Proposed shape

  • A test that sends a message near the frame cap between two nodes and asserts that heartbeat or gossip traffic in the same window is not delayed beyond the failure-detector threshold.
  • Failing that assertion is a design finding, not a test bug. If head-of-line blocking is real at the current cap, the options are a lower cap, a separate lane for control traffic, or chunking — all larger changes, and all better decided with a measurement than without one.
  • The measurement should come first regardless: what payload size, at what link speed, actually delays a heartbeat past the threshold with the default settings? That number determines whether this is urgent or theoretical.

This also connects to #1023: injecting latency would let the test construct the condition deterministically rather than relying on a payload large enough to be slow in real time.

Acceptance

  • A test measures whether control traffic is delayed behind a large payload on the same connection.
  • The payload size at which a heartbeat is delayed past the unreachability threshold is recorded on this issue.
  • If blocking is confirmed, a follow-up issue captures the design decision.

Verification status

Confirmed by reading. src/cluster/Transport.ts was read around its read path: onData pushes chunks into a per-connection decoder and dispatches complete frames in order. grep -niE "chunk|multiplex|headOfLine|priorityLane" over src/cluster/Transport.ts and src/cluster/Protocol.ts returns only TCP-level read-chunk references. No test in tests/multi-node/ or tests/unit/ sends a large payload and measures unrelated traffic. Related: #1023.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions