Add anchor-drift quality eval (graded quality marker) - #156
Open
nitininhouse wants to merge 4 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A graded quality marker for how well an agent repairs "drifted" memory after a
code change. It builds on the anchor-drift-supersede eval (a pass/fail unit test)
but measures repair quality with partial credit, so the score spreads and can
tell a good repair from a great one.
How it works
Same pipeline as the existing evals: seed memory, apply a patch, run the
update-working-memory flow, then judge and score.
bootstrap-seed.proposal.jsonseeds 14 code_verified claims across adifficulty ladder, each anchored to a real symbol.
session.patchis one realistic refactor-style change (6 files) that drifts 9of the claims in varied ways and leaves 5 fresh (some in files that change
nearby, as precision traps). The patch compiles.
rubric.jsonholds each case's difficulty, the dimensions that apply, and theground-truth expected repair.
run.tsruns the flow;evals/lib/drift-quality-scoring.tsgrades it.Drift types (the ladder)
Scoring
Analytic, partial credit per case, combined with a difficulty weight into a
0-100 score plus by-category and by-difficulty breakdowns.
Weighted toward correctness, so detecting drift without writing a good correction
does not clear the score.
Verification
and drifts exactly the 9 intended claims (7 content, 2 structural) while
leaving the 5 fresh ones byte-identical.
agent that detects drift but writes empty corrections scores below the pass
line; final score is bounded to [0, 100]; no NaN or divide-by-zero paths.
capable judge.
Notes
(gpt-4o-mini) hallucinated a preservation failure on the half-wrong case; a
stronger judge scored it correctly. The structural dimensions do not depend on
the judge.
strongest agents. The eval reports per-case difficulty so saturated cases can
be hardened over time; hardening a few cases to lower the ceiling is a natural
follow-up.
Running