Classic BM25 FTS prefilter can truncate the top BM25 match in large namespaces

[nuemaan]

Problem

The FTS prefilter added in #252 bounds candidates with ORDER BY ts_rank_cd(...) DESC LIMIT :fts_candidate_limit. When a namespace has more matching chunks than the limit, BM25 only ever sees the top slice ordered by ts_rank_cd, and that ordering disagrees with BM25 in a way that can discard the single best match.

ts_rank_cd scores on term density and coverage within a document. BM25 additionally weights by inverse document frequency, so a short chunk holding a rare query term scores very high in BM25 while ranking near the bottom under ts_rank_cd. Those are exactly the chunks the limit truncates first.

This only bites above the candidate limit, so small namespaces are unaffected. It is silent: the debug line reports candidates=2000 limit=2000 whether 2000 matched or 50000 matched and were cut to 2000.

Reproduction

Corpus of 5001 chunks. 5000 densely repeat a common term, 1 holds a rare term:

CREATE TABLE sat (
  id serial primary key,
  txt text,
  tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', coalesce(txt,''))) STORED
);
INSERT INTO sat (txt)
SELECT 'data data data data data filler ' || i FROM generate_series(1,5000) i;
INSERT INTO sat (txt) VALUES ('zebra');

Query data zebra, prefilter ordering:

total matching:        5001
zebra chunk position:  5001 of 5001     -- ts_rank_cd, dead last

Same rows through rank_rows_by_bm25 from lexical_ranker:

BM25 top 3:
   rare-zebra     score=13.2051
   common-1       score=3.8967
   common-2       score=3.8967

rare-zebra BM25 position: 1 of 5001

With the default limit of 2000 the chunk BM25 ranks first is truncated before BM25 runs.

Suggested direction

Give every query lexeme a share of the candidate budget rather than taking one global ts_rank_cd slice. Roughly limit / lexeme_count per lexeme via a lateral join, unioned and deduplicated. A lexeme matching few chunks then always contributes them, so rare terms survive. Cost is one GIN probe per lexeme instead of one overall, and the token count is already capped at 50 by _MAX_FTS_QUERY_TOKENS.

Worth surfacing saturation regardless of the ordering fix. candidate_count >= candidate_limit means the pool was truncated and recall may be reduced, which is worth a warning rather than a debug line that looks identical to the healthy case.

Scope

• packages/shared-python/shared/services/retrieval/search/channels.py, _bm25_channel
• Classic route only. Map-nav default path is unaffected.

Follows up on #195 and #252.