feat: simpler iterative retrieval (BM25 → rerank → grade → expand)

[dominikpeter]

Summary

Replaces the fan-out of fast-path / filter-task / swarm-rescue / escalate with a clean iterative loop that matches the design you sketched: BM25 with extended queries → top 20 → rerank → grader → on fail expand (more variants + semantic) → iterate up to 5 times.

Flow

preprocess ∥ detect_filter_intent          (parallel LLM calls)
seed variants = LLM synonyms + edit-distance-1 typo rewrites

for iter_n in range(max_iter):
    broad_search(variants, semantic=iter_n≥1?0.5:None)   │
                     ∥                                    │  batched via
    filter_search(intent, variants) if intent             │  Meilisearch
                                                          │  multi-search
    merge (always keep both arms) → pool
    stage-aware merge with priority → take top 20 → rerank → pin filter top-5
    grader:
        top-1 score ≥ 0.9             → pass, break
        else LLM relevance_check:
            makes_sense ∧ conf ≥ 0.7  → pass, break
    if fail: LLM generates more variants, loop

Key design choices

• Always keep the unfiltered arm: even when filter-intent fires, we still run broad search in parallel and merge both. When the intent LLM picks the wrong supplier_name value, broad_docs still has the right hits.
• Stricter grader: 0.9 score floor (was 0.7) + 0.7 LLM confidence floor (was 0.6). Marginal pools now iterate instead of early-exiting.
• Progressive semantic weight: iter 0 is BM25-only (if preprocess didn't set a ratio); iter 1+ blends in 0.5 semantic. Matches the ‘bm25 first, then hybrid, then more variants’ design.
• Stage-aware merge (_merge_with_priority): later iters can only ADD candidates — the first 5 slots are reserved for earlier-stage hits, so expansion never demotes a correctly-surfaced doc.
• Multi-search everywhere: every broad/filter/rescue call batches variants via Meilisearch /multi-search (one HTTP round-trip per stage).

Eval

3-run means on tests.eval_v2 (40 OneTrade DE base cases + Article + Supplier):

	Hit@5
baseline (main)	122/189 ±2
this PR	127/189 ±1

+5 hits, +2.7pp, first client-side win above noise floor since #18.

Latency: ~500-800ms per query on fast path (grader short-circuit). Worst case with 3 iters: ~4-6s.

Removed

• _aescalate_if_needed helper (~80 LOC) — absorbed into the loop
• Fast-path with relevance-check gate — fast-accept was bypassing filter too aggressively
• Hyde task in _aretrieve_documents — _asearch handles HyDE internally

Net: -53 LOC (361 → 308), simpler control flow.

🤖 Generated with Claude Code