Skip to content

Add anchor-drift quality eval (graded quality marker) - #156

Open
nitininhouse wants to merge 4 commits into
Autoloops:mainfrom
nitininhouse:eval/anchor-drift-quality
Open

Add anchor-drift quality eval (graded quality marker)#156
nitininhouse wants to merge 4 commits into
Autoloops:mainfrom
nitininhouse:eval/anchor-drift-quality

Conversation

@nitininhouse

Copy link
Copy Markdown
Contributor

What this is

A graded quality marker for how well an agent repairs "drifted" memory after a
code change. It builds on the anchor-drift-supersede eval (a pass/fail unit test)
but measures repair quality with partial credit, so the score spreads and can
tell a good repair from a great one.

How it works

Same pipeline as the existing evals: seed memory, apply a patch, run the
update-working-memory flow, then judge and score.

  • bootstrap-seed.proposal.json seeds 14 code_verified claims across a
    difficulty ladder, each anchored to a real symbol.
  • session.patch is one realistic refactor-style change (6 files) that drifts 9
    of the claims in varied ways and leaves 5 fresh (some in files that change
    nearby, as precision traps). The patch compiles.
  • rubric.json holds each case's difficulty, the dimensions that apply, and the
    ground-truth expected repair.
  • run.ts runs the flow; evals/lib/drift-quality-scoring.ts grades it.

Drift types (the ladder)

Difficulty Type Example
easy value minimumSelectedClaims 3 to 5; embedding 768 to 384
medium behavior scoreBm25 stops normalizing; selectRankedDocuments OR to AND
medium enum installPlatforms gains a member
hard rename tokenVariants to stemVariants (anchor breaks, must re-anchor)
hard half-wrong one threshold changes, two stay true (must preserve them)
hard removal indexScores deleted (must retire, not patch)
trap fresh stable symbols in changed files (must leave alone)

Scoring

Analytic, partial credit per case, combined with a difficulty weight into a
0-100 score plus by-category and by-difficulty breakdowns.

  • Structural dimensions (deterministic): detection, restraint, anchor_accuracy.
  • Semantic dimensions (LLM judge): correctness, completeness, preservation.
  • Semantic dimensions are gated to 0 when the claim was not superseded.

Weighted toward correctness, so detecting drift without writing a good correction
does not clear the score.

Verification

  • Fixture: all 14 anchors resolve at the base commit; the patch applies cleanly
    and drifts exactly the 9 intended claims (7 content, 2 structural) while
    leaving the 5 fresh ones byte-identical.
  • Scoring: adversarial profiles spread cleanly (28 / 59 / 73 / 79 / 100); an
    agent that detects drift but writes empty corrections scores below the pass
    line; final score is bounded to [0, 100]; no NaN or divide-by-zero paths.
  • Ran live end to end (Codex agent + judge): a correct repair scores 100 with a
    capable judge.

Notes

  • The semantic dimensions need a capable judge model. A weak judge
    (gpt-4o-mini) hallucinated a preservation failure on the half-wrong case; a
    stronger judge scored it correctly. The structural dimensions do not depend on
    the judge.
  • A top model currently scores 100, so the hard cases do not yet challenge the
    strongest agents. The eval reports per-case difficulty so saturated cases can
    be hardened over time; hardening a few cases to lower the ceiling is a natural
    follow-up.

Running

OPENAI_MODEL=<capable-model> npm run eval:anchor-drift-quality -- --judge openai --agent-model <agent-model>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants