What
Change the streaming producer's internal buffer from a Vec advanced with Vec::drain(..cursor) to a VecDeque, so advancing past consumed samples is O(1) instead of O(n).
Why
StreamingProducer currently advances its buffer with self.buffer.drain(..self.buffer_cursor) (qdp-core/src/pipeline_runner.rs:368), which memmoves the remaining elements down on every batch — O(n) per advance, paid on the streaming hot path. A VecDeque makes front-popping O(1) and avoids the repeated memmove, with no change to the data produced.
How
- Replace the backing
Vec buffer in the streaming producer (or StreamingAdapter, if E2 has landed) with a VecDeque<T>.
- Swap the
Vec::drain(..cursor) front-removal for VecDeque front operations (drain(..cursor) / pop_front) so the advance is O(1) amortized.
- Ensure capacity stays bounded across many batches (no unbounded growth or reallocation churn).
- Keep output identical to the current streaming path.
Out of scope: Producer unification (E2) if taken separately, any change to streaming correctness or output.
Acceptance criteria:
What
Change the streaming producer's internal buffer from a
Vecadvanced withVec::drain(..cursor)to aVecDeque, so advancing past consumed samples is O(1) instead of O(n).Why
StreamingProducercurrently advances its buffer withself.buffer.drain(..self.buffer_cursor)(qdp-core/src/pipeline_runner.rs:368), which memmoves the remaining elements down on every batch — O(n) per advance, paid on the streaming hot path. AVecDequemakes front-popping O(1) and avoids the repeated memmove, with no change to the data produced.How
Vecbuffer in the streaming producer (orStreamingAdapter, if E2 has landed) with aVecDeque<T>.Vec::drain(..cursor)front-removal forVecDequefront operations (drain(..cursor)/pop_front) so the advance is O(1) amortized.Out of scope: Producer unification (E2) if taken separately, any change to streaming correctness or output.
Acceptance criteria:
VecDequecapacity stays constant over 100 consecutive batches (no reallocation growth)tests/parquet_f32.rsandtests/reader.rs