Pipeline the dispatch lane: depth-4 ring, per-job timeline retirement (#151, #149)

[aravishankar-mp]

Why

Second tranche of #151 (shared with #149): amortize the ~70 µs per-submission floor by keeping
several submissions in flight. Stacked on #162.

What changed. The worker held the stream mutex across hrx_stream_dispatch plus a blocking
hrx_stream_synchronize, one job at a time — one outstanding submission, zero overlap, and
allocate_buffer queued behind every running dispatch (a finding from the map-closeout review).
The lane now keeps up to four jobs in flight: dispatch (record + flush + timeline position)
holds the stream mutex briefly, and the completion wait blocks on the stream's timeline semaphore
in bounded slices with no lock held. Every fault-tier semantic maps one-to-one; on a tier-2
wedge the worker parks forever holding every in-flight job's retained resources — the same
quarantine as before. hrx_stream_synchronize leaves the FFI; hrx_stream_flush,
hrx_stream_get_timeline_position, and hrx_semaphore_wait join it, transcribed from the pinned
headers.

A real driver behavior found on metal. The flushed batch's timeline value is assigned
asynchronously: a position read immediately after the flush occasionally still reports the
previous batch's target, and waiting on that retires a job early — observed as stale outputs
(the max-pool NaN round reading the previous submission's bytes, ~1 in 6 cold runs). Since the
stream is instance-private and dispatches are mutex-serialized, each job's tick is provably the
first timeline value observed past its predecessor's; the position read now spins on that
induction, watchdog-guarded. Six consecutive cold suite runs clean after the fix (two logged
failing runs before it).

The honest throughput result. Pipelined and sequential submission measure identically —
73.3–74.9 µs amortized vs 69.3–74.6 µs p50 across three runs each — and a batched-flush variant
(all in-flight dispatches under one hrx_stream_flush) measured 70.1 µs, also identical. The
floor is per-command driver/firmware round-trip cost inside one hardware context; no host-side
submission restructuring moves it. docs/performance.md records this, and it redirects the
remaining #151/#149 throughput work to more work per dispatch (larger envelopes, striping — #151
steps 5–6) and parallel hardware contexts (#121). The depth still pays for itself in semantics:
multiple pending events (reference parity — OpenVINO has always allowed them), submissions
overlapping host-side polling and readback, and completion waits that no longer serialize
allocate_buffer.

Coverage

All on metal (1022:17f0, v2026.08 toolchain), fail-loud under
VIRTIO_ACCEL_XDNA_REQUIRE_HARDWARE=1:

• pipelined_submissions_complete_in_order_with_their_own_payloads — four distinct submissions in
  flight for eight rounds; every completion must carry exactly its own submission's payload.
• ring_capacity_bounds_outstanding_events_and_reclaim_restores_it — deterministic replacement
  for the old depth-1 concurrent-submit test (whose Busy assertion is a race at depth 4): fills
  all four event slots, proves Busy at five, proves reclaiming one restores exactly one.
• measures_pipelined_int8_matmul_throughput (ignored, manual) — amortized per-inference cost
  over 400 completions, warmups oracle-validated.
• pending_releases_return_the_same_live_resources_for_retry now uses a new test-control
  HoldDispatch fault (bounded delay before one healthy dispatch) for a deterministic pending
  window: pipelining made the old version's implicit race losable — observed twice in logged cold
  runs before the rewrite.

Suite results: 29 passed / 0 failed (test-control) and 26 / 0 (default), including two final cold
runs of each after the last change; the shared conformance suite passes; the sequential benchmark
is unregressed (72.3 µs vs 72.1 µs on #162).

Compatibility

• No wire effect. This changes no accepted or emitted protocol bytes.

Guest-visible change: up to four submissions are now accepted before Busy (previously one). The
Busy-on-full contract, event semantics, and all fault semantics are unchanged. Buffers still
admit at most one outstanding submission each (the in-flight gates are unchanged), so pipelining
requires distinct buffer sets — the documented pattern.

Checklist

• Does not alter payload lengths, ownership, reset, error, timeout, or feature-negotiation
  behavior — the ring depth is a documented capacity, not negotiated.
• Authoritative inputs untouched.
• Public Rust API changes: none (XdnaTestFault::HoldDispatch is doc(hidden) behind
  test-control).
• No dependency changes.
• unsafe changes are confined to ffi.rs declarations transcribed from the pinned headers
  and the worker's dispatch/wait calls, each with a local safety argument; SAFETY.md's
  concurrency section is rewritten to match the new model.
• Deferred optional features remain unadvertised.

Verification

cargo fmt --all -- --check
python3 ci/check-release-policy.py
cargo clippy -p virtio-accel-xdna --all-targets --all-features --no-deps -- -D warnings   # both with and without HRX
cargo test --workspace --all-targets --all-features    # 308 passed, 0 failed (portable)
RUSTDOCFLAGS=-D warnings cargo doc -p virtio-accel-xdna --all-features --no-deps

On metal: hardware suite cold and warm in both feature configurations (logs kept for every run,
including the two pre-fix failures that motivated the timeline-induction change and the
HoldDispatch rewrite), conformance, and all three benchmarks. A peer session was using the NPU
concurrently during some runs; the reported benchmark spreads are across three runs each.

🤖 Generated with Claude Code