Skip to content

feat: Release Intelligence — tracking contracts gate release comparisons - #2

Merged
BarujaFe1 merged 14 commits into
mainfrom
feat/release-intelligence
Aug 22, 2026
Merged

feat: Release Intelligence — tracking contracts gate release comparisons#2
BarujaFe1 merged 14 commits into
mainfrom
feat/release-intelligence

Conversation

@BarujaFe1

Copy link
Copy Markdown
Owner

Problem

After a release, dashboards compare funnels across versions. If a client bug fires an event early (or twice), conversion looks different and teams ship wrong conclusions. Dashboards count events; they rarely validate what events mean first.

Solution

Release Intelligence validates versioned tracking contracts BEFORE comparing releases:

  • \data/contracts/events.yml: owner, required props, must_follow sequences, max_per_user_session, introduced_in
  • Deterministic validator (\services/instrumentation.py): missing_required_property / order_violation / duplicate_insert_id / cardinality_breach / event_drift (warning) / unknown_event (warning, never silently dropped)
  • \services/release_compare.py: raw vs trusted funnels per unique user with explicit verdicts
  • \services/journey_diff.py: added/removed session-transition edges with min support
  • Wilson CIs on activation rates; strictly observational language
  • FastAPI endpoints under /api/releases/*\ + Next.js cockpit at /releases\
  • Pre-computed snapshot for GitHub Pages (\public/data/release-intelligence.json) generated by the same backend services

The wow moment (golden fixtures, deterministic)

\
v2.3.0-buggy raw onboarding_completed +24.3% -> trusted -0.9% => instrumentation_artifact
v2.3.0-fixed activation_completed +9.1% -> trusted +9.1% => real_improvement
\\

Raw says the funnel improved after v2.3-buggy. Trusted metrics prove the client shipped broken instrumentation — onboarding_completed fired before profile_saved — and the product did not improve.

Golden scenarios G1-G6

All automated: healthy PASS / buggy order BLOCK (+24.3% raw vanishes to -0.9% trusted) / duplicate insert_id dedup+violation / real improvement survives filter / unknown event warning kept in raw. Mutation sanity performed locally: disabling order detection breaks G2; disabling dedup breaks G4 (mutations never committed).

Trade-offs

  • Raw funnel is deliberately loose (what naive dashboards count); trusted view is contract-cleaned. The contrast IS the thesis.
  • Event drift is a heuristic rate comparison vs baseline (>1.25x), documented as such.
  • Snapshot duplicates overview data for Pages robustness; single generator keeps both in sync and CI enforces byte-identical regeneration.

Evidence

  • 50 pytest tests green locally (incl. G1-G6 + API contracts), ruff clean
  • Web lint + typecheck + static build clean (/releases prerendered)
  • Local smoke: GET / -> 200, /releases/ -> 200, snapshot JSON served
  • Secret scan + PII scan clean on current tree
  • CI extended: explicit golden-scenario job, snapshot byte-determinism job, secret/PII scan job

@BarujaFe1
BarujaFe1 merged commit f82df43 into main Aug 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant