Skip to content

[Refactor] Unify PipelineIterator constructors with Source enum #1434

Description

@0lai0

What

Replace the three separate PipelineIterator constructors — new_synthetic, new_from_file, new_from_file_streaming (qdp-core/src/pipeline_runner.rs) — with a single new that takes a Source enum, so the source becomes an argument rather than a function name.

Why

The three constructors share the same normalize → build → spawn-producer-thread logic, differing only in which source they read from. Keeping three copies means every pipeline-setup change is made three times and they drift apart. A Source enum makes the source an argument rather than a function name, so the setup path exists once.

How

  • Define a Source enum in pipeline_runner.rs:
pub enum Source {
    Synthetic { total_batches: usize, seed: Option<u64> },
    File(PathBuf),
    FileStreaming(PathBuf),
}
pub fn new(engine: QdpEngine, source: Source, config: PipelineConfig) -> Result<Self>
  • new runs the shared config.normalize() / build / spawn logic and matches on source only where behavior genuinely differs (note the current file constructors also take a batch_limit — fold it into the File/FileStreaming variants or the config, and be explicit about where it lives).
  • Either keep the old new_* methods as thin deprecated wrappers that delegate to new, or update the PyO3 call sites (qdp-python/src/loader.rs) in the same PR — pick one and be consistent.
  • No producer logic change in this issue (that is E2); this is purely constructor unification.
  • Land after [Feature] Pipeline file load respects PipelineConfig.dtype #1341 (A3) where practical, to avoid merge conflicts in the same file.

Out of scope: Producer unification (E2), VecDeque buffer change (E3), any behavior change to batch production.

Acceptance criteria:

  • One test per Source variant constructed via PipelineIterator::new
  • Old new_* paths still work (as wrappers) or all call sites updated in-PR
  • All existing pipeline tests pass with no behavior change

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions