feat: Release Intelligence — tracking contracts gate release comparisons - #2
Merged
Conversation
…buggy / v2.3 fixed
…trumentation health, journey diff
…n/secret gates + README section
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After a release, dashboards compare funnels across versions. If a client bug fires an event early (or twice), conversion looks different and teams ship wrong conclusions. Dashboards count events; they rarely validate what events mean first.
Solution
Release Intelligence validates versioned tracking contracts BEFORE comparing releases:
The wow moment (golden fixtures, deterministic)
\
v2.3.0-buggy raw onboarding_completed +24.3% -> trusted -0.9% => instrumentation_artifact
v2.3.0-fixed activation_completed +9.1% -> trusted +9.1% => real_improvement
\\
Raw says the funnel improved after v2.3-buggy. Trusted metrics prove the client shipped broken instrumentation — onboarding_completed fired before profile_saved — and the product did not improve.
Golden scenarios G1-G6
All automated: healthy PASS / buggy order BLOCK (+24.3% raw vanishes to -0.9% trusted) / duplicate insert_id dedup+violation / real improvement survives filter / unknown event warning kept in raw. Mutation sanity performed locally: disabling order detection breaks G2; disabling dedup breaks G4 (mutations never committed).
Trade-offs
Evidence