Skip to content

Add or document a reusable full-pipeline runner for mixed RAG adapters #6

@laynepenney

Description

@laynepenney

Finding

The adapter pattern works for a mixed retrieval + generation customer flow, but there is no reusable runner that wires the two surfaces together.

Validation

A customer-shaped probe passed with three adapter shapes:

  1. retrieval-only: RetrievalAdapter + run_retrieval_category
  2. generation-only: GenerationAdapter + run_generation_category
  3. mixed/RAG: retrieve candidates, convert them to context manually, then call GenerationAdapter.generate

The mixed path works, but it requires hand-rolled orchestration in application code. The repository's examples/full-pipeline/run.py follows the same manual pattern.

Rough edge

Customers with RAG pipelines likely expect a reusable full-pipeline runner that:

  • calls retrieval
  • passes retrieved context to generation
  • records retrieval and generation per-fixture details together
  • optionally runs reviewer predicates or judges
  • aggregates both retrieval and generation metrics into one EvalResult/report card shape

Today they can build this manually, but the first integration has more glue than retrieval-only or generation-only paths.

Recommendation

Add either:

  • a run_full_pipeline_category(...) helper, or
  • a docs section explicitly naming the manual orchestration pattern as the supported v0.1 approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions