Finding
The adapter pattern works for a mixed retrieval + generation customer flow, but there is no reusable runner that wires the two surfaces together.
Validation
A customer-shaped probe passed with three adapter shapes:
- retrieval-only:
RetrievalAdapter + run_retrieval_category
- generation-only:
GenerationAdapter + run_generation_category
- mixed/RAG: retrieve candidates, convert them to context manually, then call
GenerationAdapter.generate
The mixed path works, but it requires hand-rolled orchestration in application code. The repository's examples/full-pipeline/run.py follows the same manual pattern.
Rough edge
Customers with RAG pipelines likely expect a reusable full-pipeline runner that:
- calls retrieval
- passes retrieved context to generation
- records retrieval and generation per-fixture details together
- optionally runs reviewer predicates or judges
- aggregates both retrieval and generation metrics into one
EvalResult/report card shape
Today they can build this manually, but the first integration has more glue than retrieval-only or generation-only paths.
Recommendation
Add either:
- a
run_full_pipeline_category(...) helper, or
- a docs section explicitly naming the manual orchestration pattern as the supported v0.1 approach.
Finding
The adapter pattern works for a mixed retrieval + generation customer flow, but there is no reusable runner that wires the two surfaces together.
Validation
A customer-shaped probe passed with three adapter shapes:
RetrievalAdapter+run_retrieval_categoryGenerationAdapter+run_generation_categoryGenerationAdapter.generateThe mixed path works, but it requires hand-rolled orchestration in application code. The repository's
examples/full-pipeline/run.pyfollows the same manual pattern.Rough edge
Customers with RAG pipelines likely expect a reusable full-pipeline runner that:
EvalResult/report card shapeToday they can build this manually, but the first integration has more glue than retrieval-only or generation-only paths.
Recommendation
Add either:
run_full_pipeline_category(...)helper, or