consomme: autotune per-connection TCP ring buffers by benhillis · Pull Request #3597 · microsoft/openvmm

benhillis · 2026-05-29T20:47:21Z

Grows per-connection TCP rings in place from initial up to max (TcpBufferBounds) based on pressure, instead of a fixed size. The default now autotunes (16 KiB → 4 MiB), so idle/short connections start cheaper than the old fixed 256 KiB while bulk flows can ramp to a few MiB.

What changed

ConsommeParams exposes per-connection tcp_rx_buffer/tcp_tx_buffer as TcpBufferBounds { initial, max } so embedders can pick bounds for their workload.
Ring::resize: grow a power-of-two ring in place, preserving the [head, tail) view. rx grow is gated on the assembler + rx buffer being empty.
rx window scale derived from rx.max so the advertised window can grow without re-negotiating at SYN.
poll_socket_backend: double the tx ring when it fills the host read loop; double the rx ring + window when it hit ≥75% then drained (forcing an ACK so the peer picks up the larger window). Capped at max. New tx_buffer_grows/rx_buffer_grows counters.
DEFAULT_TCP_BUFFER_BOUNDS now 16 KiB → 4 MiB (was fixed 256 KiB).

Testing

cargo nextest run -p consomme passes (incl. new ring::resize tests); cargo clippy --release -p consomme --all-targets clean.

burette network benchmark (--backend consomme --nic vmbus, N=5, warm VM), old fixed 256 KiB vs autotune 64 KiB → 4 MiB:

	Baseline	Autotune	Δ
TCP tx	5.40	9.94 Gbps	+84%
TCP rx	3.86	5.21 Gbps	+35%

A fixed-4 MiB control matches autotune (9.34 / 5.27), so the win is from raising the 256 KiB ceiling — autotune gets it while still starting small.

github-actions · 2026-05-29T20:49:07Z

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

benhillis · 2026-05-29T20:50:55Z

Oops this was supposed to be an RFC / draft.

Copilot

Pull request overview

This PR adds Linux-style TCP autotuning to the Consomme user-mode TCP stack by replacing fixed per-connection TX/RX ring sizes with bounded growth (initial → max) and resizing rings in-place based on observed buffer pressure, while keeping the default behavior equivalent to the prior fixed 256 KiB buffers.

Changes:

Introduces TcpBufferBounds { initial, max } on ConsommeParams and wires those bounds into TCP connection initialization.
Adds Ring::resize() to grow ring buffers while preserving the in-order [head, tail) byte range, plus unit tests for resize behavior.
Implements per-connection autotune in the socket backend poll loop and adds inspect counters for TX/RX buffer growth.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
vm/devices/net/net_consomme/consomme/src/tcp/ring.rs	Adds `Ring::resize()` and new unit tests covering resize across wrap/no-wrap scenarios.
vm/devices/net/net_consomme/consomme/src/tcp/assembler.rs	Makes `Assembler::is_empty()` available outside tests to gate safe RX ring growth.
vm/devices/net/net_consomme/consomme/src/tcp.rs	Plumbs buffer bounds into connection creation, derives window scale from RX max, and grows TX/RX rings opportunistically with new inspect counters.
vm/devices/net/net_consomme/consomme/src/lib.rs	Exposes TCP RX/TX buffer bounds on `ConsommeParams` with defaults preserving prior fixed sizing.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

Address review feedback on microsoft#3597: - Clarify that the rx ring can round above rx_buffer_max (power-of-two) and that TcpBufferBounds is capped by u16::MAX without window scaling. - Add tests covering buffer-bounds normalization, that the derived rx window scale reaches max, and that the tx ring autotunes (grows and caps at max) under host-side backpressure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Address review feedback on microsoft#3597: note the new per-connection ring buffer autotune defaults (16 KiB initial, 4 MiB max) and how embedders override them via tcp_rx_buffer/tcp_tx_buffer on ConsommeParams. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

github-actions · 2026-06-03T18:10:46Z

1 Petri tests failed (0 unstable)

github-actions · 2026-06-03T20:30:10Z

4 Petri tests failed (4 unstable)

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Address review feedback on microsoft#3597: - Clarify that the rx ring can round above rx_buffer_max (power-of-two) and that TcpBufferBounds is capped by u16::MAX without window scaling. - Add tests covering buffer-bounds normalization, that the derived rx window scale reaches max, and that the tx ring autotunes (grows and caps at max) under host-side backpressure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address review feedback on microsoft#3597: note the new per-connection ring buffer autotune defaults (16 KiB initial, 4 MiB max) and how embedders override them via tcp_rx_buffer/tcp_tx_buffer on ConsommeParams. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Grow per-connection TCP rings in place from `initial` up to `max` (`TcpBufferBounds`) based on observed pressure, instead of using a fixed size. The default now autotunes (16 KiB -> 4 MiB), so idle/short connections start cheaper than the old fixed 256 KiB while bulk flows can ramp to a few MiB. What changed: - `ConsommeParams` exposes per-connection `tcp_rx_buffer`/`tcp_tx_buffer` as `TcpBufferBounds { initial, max }` so embedders can pick bounds for their workload. `DEFAULT_TCP_BUFFER_BOUNDS` is now 16 KiB -> 4 MiB (was a fixed 256 KiB). - `Ring::resize`: grow a power-of-two ring in place, preserving the `[head, tail)` view without staging through a temporary buffer. Grow-only (same-capacity resizes are a no-op). rx grow is gated on the assembler + rx buffer being empty. - rx window scale is derived from `rx.max` so the advertised window can grow without re-negotiating at SYN. Window/ring sizing accounts for the power-of-two rounding and the `u16::MAX` cap when window scaling is unavailable. - `poll_socket_backend`: double the tx ring when it fills the host read loop (re-reading the host socket into the freshly grown ring within the same poll), and double the rx ring + window when it hit >=75% then drained (forcing an ACK so the peer picks up the larger window). Both are capped at `max`. New `tx_buffer_grows`/`rx_buffer_grows` counters. Tests: - New `ring::resize` unit tests and end-to-end autotune tests covering tx/rx growth, capping at `max`, and the derived rx window scale. - Test host-write helpers poll consomme concurrently so large writes can't deadlock when consomme is the only socket reader. Docs: - The Guide's consomme backend page documents the new default bounds, autotune behavior, and tuning knobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

github-actions · 2026-06-06T01:52:48Z

1 Petri tests failed (0 unstable)

benhillis requested a review from a team as a code owner May 29, 2026 20:47

Copilot AI review requested due to automatic review settings May 29, 2026 20:47

Copilot started reviewing on behalf of benhillis May 29, 2026 20:47 View session

github-actions Bot added the unsafe Related to unsafe code label May 29, 2026

benhillis marked this pull request as draft May 29, 2026 20:50

benhillis changed the title ~~consomme: autotune per-connection TCP ring buffers~~ RFC: consomme: autotune per-connection TCP ring buffers May 29, 2026

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/ring.rs

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/ring.rs Outdated

benhillis changed the title ~~RFC: consomme: autotune per-connection TCP ring buffers~~ consomme: autotune per-connection TCP ring buffers May 29, 2026

benhillis marked this pull request as ready for review May 29, 2026 22:10

Copilot AI review requested due to automatic review settings May 29, 2026 22:10

Copilot started reviewing on behalf of benhillis May 29, 2026 22:10 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/ring.rs

Comment thread vm/devices/net/net_consomme/consomme/src/tcp.rs Outdated

Copilot AI review requested due to automatic review settings June 3, 2026 04:09

Copilot started reviewing on behalf of benhillis June 3, 2026 04:09 View session

benhillis mentioned this pull request Jun 3, 2026

consomme: make per-connection TCP buffer sizes configurable #3595

Closed

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp.rs Outdated

Comment thread vm/devices/net/net_consomme/consomme/src/tcp.rs Outdated

Comment thread vm/devices/net/net_consomme/consomme/src/lib.rs

Comment thread vm/devices/net/net_consomme/consomme/src/tcp.rs

benhillis requested a review from Copilot June 3, 2026 15:08

Copilot started reviewing on behalf of benhillis June 3, 2026 15:08 View session

Copilot AI reviewed Jun 3, 2026

benhillis requested a review from Copilot June 3, 2026 15:25

Copilot started reviewing on behalf of benhillis June 3, 2026 15:25 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp.rs Outdated

Comment thread vm/devices/net/net_consomme/consomme/src/lib.rs

github-actions Bot added the Guide label Jun 3, 2026

Copilot AI review requested due to automatic review settings June 3, 2026 16:47

Copilot started reviewing on behalf of benhillis June 3, 2026 16:47 View session

github-actions Bot removed the unsafe Related to unsafe code label Jun 3, 2026

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/tests.rs Outdated

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/tests.rs Outdated

benhillis requested a review from Copilot June 5, 2026 15:38

Copilot started reviewing on behalf of benhillis June 5, 2026 15:38 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/tests.rs

Comment thread vm/devices/net/net_consomme/consomme/src/tcp/tests.rs

Copilot AI review requested due to automatic review settings June 6, 2026 00:02

benhillis force-pushed the user/benhillis/consomme-tcp-buffer-autotune branch from 33f6ea6 to 9ec22d5 Compare June 6, 2026 00:02

Copilot started reviewing on behalf of benhillis June 6, 2026 00:02 View session

benhillis force-pushed the user/benhillis/consomme-tcp-buffer-autotune branch from 9ec22d5 to 200c4bc Compare June 6, 2026 00:05

Copilot AI reviewed Jun 6, 2026

View reviewed changes

jstarks approved these changes Jun 9, 2026

View reviewed changes

Conversation

benhillis commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Testing

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

benhillis commented May 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benhillis commented May 29, 2026 •

edited

Loading