feat(OBS-004): alert when a universe silently loses mapped symbols

[DoRmAmMu1997]

Closes #119.

Problem

When Dhan's instrument master stops listing a symbol, refresh_universe_files() marks it missing_security_id and mapped_only() filters it out of every scan. That is correct — we cannot fetch candles for a security id we do not have. What was missing is that nothing said so:

• universe_status() had exactly one non-test caller: ui/status_panel.py, the interactive sidebar.
• backend/jobs/run_daily_scan.py emitted no mapping signal at all.
• backend/notifications/ never referenced mapping_status.

So in production — Render's cron refreshes universes and scans with nobody watching a sidebar — a universe could shrink indefinitely. It already had: ~3% of the Hemant Good 200 list was unscanned, and two more names dropped out mid-audit without a sound.

Worth being clear, because it shaped the design: the drop-outs were not a bug. JBCHEPHARM and GUJGASLTD were genuinely absent from the 2026-08-24 master by symbol and company name, with the snapshot itself intact at 213,213 rows. This is a real vendor event the system should report, not prevent.

Design

Three pieces, split by requirement. collect_universe_health() and detect_mapping_regressions() are pure and trivially testable; check_universe_health(session) is the stateful one, because "worse than last time" needs a durable baseline.

A table, not a file. The Render daily-scan cron runs on an ephemeral filesystem with no disk — render.yaml attaches the disk to the web service only. A file-based baseline could never fire the alert in the one environment that needs it. Rows are append-only rather than upserted, because the question after "GUJGASLTD dropped out" is always "when?".

Two rules keep the alert credible enough that nobody mutes the channel:

1. A universe with no previous row never regresses — otherwise the first run alerts on every pre-existing unmapped symbol.
2. Only an increase counts — steady-state damage stays quiet, and recovery is not an alert either.

Only the alerting path owns the baseline. This is the subtle one. Whoever writes the baseline defines what "last time" means, so if the morning prefetch also recorded a snapshot, a symbol dropping out at 09:00 would already be baseline by the evening and the alert would never fire. The prefetch therefore logs health without persisting; the daily job reads the baseline, compares, and only then records today's — which is exactly what makes the alert fire once.

Bounded by construction. MAX_REPORTED_SYMBOLS = 25 caps both the stored JSON and the alert text, so a badly broken CSV cannot write an unbounded blob to Postgres or a multi-page Telegram message. Symbols and counts only — never prices — and the text still goes through redact_text.

Failure posture. Every entry point catches broadly and degrades to no-alert. A universe CSV that will not parse is a reason to warn, never a reason to skip the night's scan; the check can't change the job's exit code.

What an operator sees

universe_health_checked per universe on every run (rows / mapped / unmapped), a universe_mapping_regressed warning when something got worse, a printed job line, and a new alert block:

Universe warnings:
  - hemant_good_200: 6 -> 8 unmapped (+2); GUJGASLTD, JBCHEPHARM

That block renders even at the ALERT-002 summary-only content level — a shrinking universe is a warning about the integrity of this scan, not a per-stock result.

Acceptance criteria from the issue

Criterion	Test
A universe losing a symbol produces exactly one alert on the run where it happens	test_check_alerts_exactly_once_when_a_symbol_drops_out (runs the check three times)
A steady-state universe with pre-existing unmapped symbols produces none	test_steady_state_and_recovery_do_not_regress, test_no_baseline_never_regresses
The daily job's events carry the per-universe counts	test_log_universe_health_emits_one_event_per_universe

Verification

• pytest -q --cov=... --cov-fail-under=89 — 2052 passed, 1 skipped, coverage 90.07% (up from 89.97%)
• backend/data_quality/universe_health.py — 93% from its own tests
• tests/test_scan_storage_migrations.py — passes, so alembic upgrade head builds exactly Base.metadata
• ruff check, compileall, bandit — clean; mypy clean (261 files, run as --python-version 3.13; the pinned 3.11 target cannot run on this machine — numpy 2.5.2 vs the pinned 2.4.6 — so CI is the authority there)
• git diff origin/main HEAD -- constraints.txt pyproject.toml — empty

Per AGENTS.md §7 the Alembic migration ships in the same commit as the ORM change, and both hardcoded table-name sets in tests/test_scan_storage_migrations.py are updated.

Alternatives rejected

Recorded in full in docs/architecture/obs-004-universe-health-alerts.md §5 — briefly: an absolute threshold (every universe needs its own tuned number, and slow drift stays invisible), app_config as the store (it is read wholesale by apply_config_overrides(); job state could be applied as a setting), upserting one row per universe (throws away the "when?" answer for no real saving), and failing the scan on a shrink (wrong severity — a delisting is normal and the remaining scan is still valid).

🤖 Generated with Claude Code