Openconceptlab/ocl_issues#2505 | Fix US/British spelling mismatch in semantic concept search

[snyaggarwal]

Refs OpenConceptLab/ocl_issues#2505 (Problem B only — Problem A still open)
Refs OpenConceptLab/ocl_issues#2511 (Lexical Variant Dictionary infrastructure)

Heads up — approach changed mid-flight

Sunny — apologies for the redirect. After deeper review of #2505 with @paynejd, we landed on a substantially different architecture and felt it was faster to push the new approach directly than to iterate via comments. Your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept — only the variant source changed from inline regex to a dictionary-driven lookup. Happy to walk through the diff on a call if useful.

Three reasons for the redirect:

1. False positives in the original regex list. hem/haem matches themselves, anthem, hemisphere, hemp, hemlock, remember — hard to fix defensively in regex without enumerating exclusions for every English word.
2. Reusability. The fix should be available beyond ConceptFuzzySearch. Standard /concepts/?q=… has the same gap, and upcoming abbreviation-expansion work needs the same primitive. A dictionary-as-Source approach makes that reuse trivial.
3. Eat our own dog food. Storing dictionaries as OCL Sources gives versioning, release management, locale support, editability through the OCL UI, and discoverability for free — and aligns with how UMLS / SNOMED / OBO model lexical variants.

Tradeoff worth flagging

This approach puts pressure on oclapi2 to do caching well. The MVP scale is trivial (~40 entries, in-memory dict), but as dictionary types and sizes grow — especially abbreviation dictionaries (UMLS LRABR is ~250k rows) and once variants are enabled by default on high-QPS endpoints — the per-token DB lookup will need to graduate beyond the MVP's module-level cache. Likely Redis-backed token-indexed cache for >1k-entry dictionaries; ES-backed lookup for very large ones. Plan to revisit when (a) a dictionary exceeds ~5k entries, or (b) variants are enabled by default on a high-QPS endpoint. Captured in #2511.

What this PR does

• New helper: core/common/lexical_variants.py — get_lexical_variants(text, source_uri) and get_variant_terms(text, source_uri). Tokenizes input (lowercase + whitespace + ASCII punctuation strip), looks each token up against the dictionary's Names, returns sibling Names from the matching Concept. Module-level cache keyed on (source_uri, version). Returns [] and never raises if the dictionary is missing — graceful degradation.
• Seed data: core/common/data/lexical-variants-en.json — OCL bulk import file creating OCL/lexical-variants-en (Source + 43 vetted Concepts + Source Version v1.0). Each Concept has en-US and en-GB Names with name_type=Fully Specified. False-positive-prone substring entries dropped.
• Wired into $match: MetadataToConceptsListView reads variants from request body, passes the resolved URI (or None) to ConceptFuzzySearch.search(..., variants_repo=...). The kNN sub-query construction and rescore expansion are unchanged in shape; only the variant source changed.
• Empty-synonyms rescore crash fix: preserved from your initial commit.
• Removed: get_spelling_variant() from core/common/utils.py.

API: variants request body field

Lexical variant expansion is OFF by default. Clients opt in. Same shape will apply to standard concept search when that wiring lands as a follow-up.

Value	Behavior
absent / null / false / "false" / "0"	disabled (default)
true / "true" / "1"	enabled, default dictionary (OCL/lexical-variants-en)
URI string (e.g. "/orgs/myorg/sources/my-dictionary/")	enabled, that dictionary

Tests

core/common/tests.py LexicalVariantsTest covers:

• Tokenization
• Multi-token expansion
• Cache lifecycle (per-source-version) + invalidation
• Missing-source graceful degradation
• Regression: themselves, anthem, hemisphere, hemp, hemlock, remember return no variants

Deployment

The dictionary Source has already been seeded on prod (OCL/lexical-variants-en v1.0, 43 concepts, public read) so the wiring will work as soon as this PR merges and deploys. For dev / qa / staging environments, run:

ocl --server ocl-dev import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-qa import file --wait core/common/data/lexical-variants-en.json
ocl --server ocl-staging import file --wait core/common/data/lexical-variants-en.json

Verify with: ocl repo get OCL lexical-variants-en (should show 43 concepts, source_type: "Lexical Variants", extras.dictionary_kind: "lexical_variant", version v1.0 released).

The helper degrades gracefully (returns [], never raises) if the Source doesn't exist yet — so a deploy gap on any environment won't break search.

Out of scope (deferred follow-ups)

• ?variants=... on standard concept search (ConceptListView) — touches CustomESSearch / get_raw_search_string layer, smaller blast radius as a separate PR
• $lexical-variants HTTP operation
• Abbreviation dictionary type (Mapping-based)
• Multi-dictionary composition
• Problem A from #2505 (CIEL bridge exact-match scoring via apply_score() highlight) — entirely separate work; this PR no longer auto-closes that part of the ticket

Architecture / design references

• Full design proposal: Lexical Variant Dictionaries — UMLS / SNOMED / OBO precedents, dictionary-type modeling decisions, tokenization-first reasoning, multi-dictionary roadmap
• Infrastructure tracking ticket: #2511 — Phase 2/3 work (standard search wiring, $lexical-variants operation, abbreviation dictionary type, caching strategy, etc.)

[paynejd]

@snyaggarwal — heads up, I pushed two commits on top of yours that redirect this PR to a dictionary-driven approach (full rationale in the updated description above + the proposal doc).

TL;DR — your kNN sub-query construction, rescore expansion, and the empty-synonyms crash fix are all kept. The variant source changed from inline regex to a dictionary lookup against ocl/lexical-variants-en (a new OCL Source seeded by bulk import). Variant expansion is now off by default; clients opt in via variants: true (or a custom dictionary URI) in the request body. We'll need follow up work to utilize this in Mapper (expose in project config?).

Apologies for the in-flight redirect — the false-positive cases in the regex list (themselves, anthem, hemisphere, hemp, etc. all matching hem/haem) plus the realization that the same primitive will be needed for upcoming synonym/abbreviation/keyword-expansion evaluation that we're starting pushed me to the dictionary-as-Source approach. Phase 2/3 work tracked in #2511.

Happy to walk through the diff or rework anything that doesn't sit right.

[snyaggarwal]

Two missing things:

1. Project config to enable it ("use_lexical_variants") and for core users only
2. Use Redis caching with TTL

@snyaggarwal to take on this

[paynejd]

Reviewing the latest revision after the redis-cache refactor and use_lexical_variants field addition. Four things to address before merge:

1. LEXICAL_VARIANTS_CACHE_TIMEOUT env override is broken — os.environ.get returns a string when set, but django-redis's cache.set(timeout=...) expects an int. Suggestion inline.

2. use_lexical_variants MapProject flag is persistence-only. The field exists, the serializer surfaces it, but MetadataToConceptsListView.filter_queryset still only reads request.data.get('variants') — so the flag has no effect at $match time until oclmap is updated to read the project config and pass variants: true per request. That's consistent with how encoder_model flows (client passes it per call), so the field is correct as persistence; just call out the dependency in the PR body and file an oclmap follow-up ticket. Otherwise reviewers / Mapper users will reasonably assume flipping the flag does something on its own.

3. Fixture path changed. Sunny moved core/common/data/lexical-variants-en.json → core/lookup_fixtures/lexical-variants-en.json, but (a) import_lookup_values.py has a hardcoded list and does not auto-load this file, so the lookup_fixtures/ placement falsely implies auto-load, and (b) the PR body's deploy commands still reference the old core/common/data/... path — anyone running them on dev/qa/staging gets a 404. Either move the file back, or wire it into import_lookup_values and update the PR body.

4. Rebase needed. PR is CONFLICTING (8 commits behind master). Conflicts are in core/map_projects/{models,serializers,tests/tests}.py from master's prompt_output_locale config addition (c883bd0). Should be a clean resolve — just append use_lexical_variants after prompt_output_locale in each CONFIGURATION_FIELDS / serializer field list / prepare_object loop. Verify migration 0036_mapproject_use_lexical_variants.py doesn't collide with master's migration numbering after rebase.

The redis-cache refactor itself looks good — keying on (source.uri, version) cleanly invalidates released-version edits. HEAD edits will be stale up to the TTL (4 days default), which is fine for a 43-entry vetted dictionary; worth a one-line comment so future maintainers know. The is_core_user gate matches the encoder_model precedent.