Skip to content

chore(retrieval): swap rank-bm25 for bm25s in BEIR harness, drop dead prod dep - #26

Merged
sanjeevafk merged 1 commit into
mainfrom
chore/bm25s-swap
Sep 4, 2026
Merged

chore(retrieval): swap rank-bm25 for bm25s in BEIR harness, drop dead prod dep#26
sanjeevafk merged 1 commit into
mainfrom
chore/bm25s-swap

Conversation

@sanjeevafk

Copy link
Copy Markdown
Owner

rank-bm25 had zero production usage (prod lexical is Postgres ts_rank) and is unmaintained since 2021. This swaps the BEIR harness to maintained bm25s and removes the dead prod dep.

Parity gate (scifact 300/25, before -> after):

  • Lexical NDCG 0.8025 -> 0.7905, recall unchanged at 0.84
  • Top-1 agreement 23/25, mean top-10 overlap 90.8% (delta is idf-smoothing minutiae, no systematic shift)
  • Lexical p50 1.34ms -> 0.11ms (12x, compounds at Tier-1 scale)

Changes:

  • scripts/benchmark_beir.py: bm25s.BM25().index() on identical .lower().split() tokens (no stemming, comparability preserved); get_scores call site API-identical
  • pyproject.toml: rank-bm25 removed from prod deps, bm25s>=0.3.0 added to dev extras
  • api/requirements.txt: rank-bm25 removed (slims prod image)

Note: prior results.json lexical columns are now non-comparable history; future runs anchor to 0.7905.

Verification: 177/177 pytest pass, ruff + compileall clean.

… prod dep

Lexical parity on scifact 300/25: NDCG 0.8025 -> 0.7905, recall 0.84
unchanged, top-1 agreement 23/25. Lexical p50 1.34ms -> 0.11ms.
rank-bm25 had zero production usage (prod lexical is Postgres
ts_rank); bm25s moves to dev extras for the benchmark harness only.
@sanjeevafk
sanjeevafk merged commit 855009d into main Sep 4, 2026
7 checks passed
@sanjeevafk
sanjeevafk deleted the chore/bm25s-swap branch September 4, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant