Pipeline the dispatch lane: depth-4 ring, per-job timeline retirement (#151, #149)

crates/virtio-accel-xdna/README.md

@@ -11,7 +11,8 @@ build time; a compile-only unsupported-runtime placeholder elsewhere.
 In a `va_xdna` build it runs the full `Accelerator`
 lifecycle — device/stream owner, `hrx_buffer` primitives (persistent mapping, range
 flush/invalidate, release), and a serialized dispatch worker bridging
-`hrx_stream_dispatch`/`synchronize` to a latched nonblocking `poll_event`. `load_program` accepts
+`hrx_stream_dispatch`/timeline-semaphore completion to a latched nonblocking `poll_event`, with
+up to four submissions in flight per instance. `load_program` accepts
 the crate-local precompiled artifact format directly, and a TOSA artifact by admitting it and
 compiling it with the bounded aiecc helper subprocess (`compiler/xdna_compile.py`, run under the
 pinned toolchain venv in a cleared environment, content-addressed in a cache). The compilable TOSA
@@ -104,10 +105,10 @@ RESCALE applies its signed INT8 output zero point only after exact 64-bit multip
 ## Completion and fault model
 
 One worker serializes each instance's accepted submissions. Finite timeouts are rejected before
-admission because HRX exposes no cancellation primitive. A definite dispatch/synchronize failure
+admission because HRX exposes no cancellation primitive. A definite dispatch or completion-wait failure
 becomes a stable terminal `Failed` event and poisons that backend instance (device-loss tier 1);
 the event and its buffers can still be released normally. A 120-second userspace watchdog, longer
-than the kernel's 60-second NPU TDR, detects a synchronize call that never returns (tier 2). In that
+than the kernel's 60-second NPU TDR, detects a dispatch or completion wait that never finishes (tier 2). In that
 case `poll_event` reports `DeviceLost`, the event remains pending and cannot be released, and the
 host must discard the backend instance. The detached worker retains the stream, executable, and
 buffer allocations so discarding cannot free native memory that HRX might still touch.

crates/virtio-accel-xdna/SAFETY.md

@@ -68,9 +68,21 @@ the buffer's `TRANSFER_DESTINATION`/`TRANSFER_SOURCE` usage.
 ## Concurrency and the dispatch worker
 
 The HRX stream is not safe for concurrent use, so all stream access is serialized by the `Lane`
-stream mutex: `allocate_buffer` locks it briefly, and the worker holds it across
-`hrx_stream_dispatch` + `hrx_stream_synchronize`. `Stream` is `unsafe impl Send` (moved to the
-worker, only ever dereferenced under that mutex).
+stream mutex: `allocate_buffer` locks it briefly, and the worker locks it briefly per dispatch
+(`hrx_stream_dispatch` + `hrx_stream_flush` + the timeline-position read). The completion wait
+holds no lock at all: it blocks on the stream's timeline semaphore (`hrx_semaphore_wait` in
+bounded slices), a standalone synchronization object that is valid while the lane retains the
+stream and safe to wait on while another thread holds the stream mutex — so `allocate_buffer`
+never queues behind a running dispatch. `Stream` is `unsafe impl Send` (moved to the worker, only
+ever dereferenced under that mutex).
+
+Up to the ring depth (four) submissions are in flight on the stream at once; the stream executes
+them in order and the worker retires them oldest-first. Each job's completion tick is the first
+timeline value observed past its predecessor's: the flushed batch's value is assigned
+asynchronously, so a position read on an unlucky schedule still reports the previous batch's
+target, and waiting on that retires early (observed on metal as stale outputs before the
+induction-anchored spin was added). The stream is instance-private and dispatches are serialized
+under the mutex, which is what makes the induction sound.
 
 `submit` validates the bindings in one pass (a nonempty, within-limit list; a queue, program, and
 every buffer from one context; unique slots via a 256-bit occupancy mask; per-slot access; ranges;
@@ -88,18 +100,23 @@ dereferenced only on the worker while the stream mutex is held. The in-flight ga
 those `BufferInner` Arcs, not caller structs: the contract requires the caller keep handles *alive*
 until the event is terminal, not address-stable, so a caller may legally move an `XdnaBuffer`
 mid-flight; the shared allocation keeps both handle and gate valid regardless. The worker, per job,
-dispatches, synchronizes,
+dispatches, waits for the job's timeline tick,
 `invalidate_range`s each output, clears the `in_flight` gates, and latches the event's terminal
-state exactly once. While a buffer's gate is set, `write_buffer`/`read_buffer`/`free_buffer` reject
+state exactly once — releasing the job's ring capacity before the terminal state becomes
+observable, so a caller that polls completion and immediately resubmits never bounces off a stale
+count. While a buffer's gate is set, `write_buffer`/`read_buffer`/`free_buffer` reject
 with `Busy`, so no host access or release races the device.
 
-Finite timeouts are rejected before admission (no cancellation exists at any layer). A synchronize
+Finite timeouts are rejected before admission (no cancellation exists at any layer). A dispatch or wait
 error latches the event `Failed` (a normal terminal state — the kernel TDR has quiesced the device)
-and then poisons the instance, which refuses further work with `DeviceLost`. `poll_event` reads the
-latched atomic state without touching HRX. The worker arms a 120-second watchdog immediately before
-dispatch, longer than the kernel's 60-second NPU TDR. If HRX still has not returned, the watchdog
-poisons the lane but deliberately leaves the accepted event pending and every gate armed: there is
-no trustworthy completion boundary. `poll_event` then reports `DeviceLost`; event release remains
+and then poisons the instance, which refuses further work with `DeviceLost`; jobs already accepted
+behind the failure latch `Failed(DeviceLost)` without touching the dead stream. `poll_event` reads
+the latched atomic state without touching HRX. The worker arms a 120-second watchdog around each
+dispatch and each completion wait, longer than the kernel's 60-second NPU TDR. If the boundary is
+declared lost, the watchdog poisons the lane but deliberately leaves the accepted events pending
+and every gate armed: there is no trustworthy completion boundary. The worker then parks forever
+holding every in-flight job's retained resources and the stream — the quarantine — and `Drop`
+detaches it. `poll_event` then reports `DeviceLost`; event release remains
 retryably rejected as `Busy`; discarding the backend enters the quarantine described above.
 `EVENT_CANCELLATION` is not advertised.
 

crates/virtio-accel-xdna/src/ffi.rs

@@ -71,6 +71,8 @@ pub(crate) type hrx_stream_t = *mut hrx_stream_s;
 pub(crate) type hrx_buffer_t = *mut hrx_buffer_s;
 /// `hrx_executable_t` — refcounted executable handle.
 pub(crate) type hrx_executable_t = *mut hrx_executable_s;
+pub(crate) enum hrx_semaphore_s {}
+pub(crate) type hrx_semaphore_t = *mut hrx_semaphore_s;
 
 /// Borrowed byte span (`hrx_const_byte_span_t`).
 #[repr(C)]
@@ -87,6 +89,14 @@ pub(crate) struct hrx_string_view_t {
 }
 
 /// Dispatch grid configuration (`hrx_dispatch_config_t`); the amdxdna path uses {1,1,1}/{1,1,1}/0.
+/// `hrx_timeline_point_t` (hrx_runtime.h:250): a stream-timeline completion marker.
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub(crate) struct hrx_timeline_point_t {
+    pub(crate) semaphore: hrx_semaphore_t,
+    pub(crate) value: u64,
+}
+
 #[repr(C)]
 pub(crate) struct hrx_dispatch_config_t {
     pub workgroup_count: [u32; 3],
@@ -222,5 +232,14 @@ unsafe extern "C" {
         binding_count: usize,
         flags: u32,
     ) -> hrx_status_t;
-    pub(crate) fn hrx_stream_synchronize(stream: hrx_stream_t) -> hrx_status_t;
+    pub(crate) fn hrx_stream_flush(stream: hrx_stream_t) -> hrx_status_t;
+    pub(crate) fn hrx_stream_get_timeline_position(
+        stream: hrx_stream_t,
+        position: *mut hrx_timeline_point_t,
+    ) -> hrx_status_t;
+    pub(crate) fn hrx_semaphore_wait(
+        semaphore: hrx_semaphore_t,
+        value: u64,
+        timeout_ns: u64,
+    ) -> hrx_status_t;
 }