fix(metrics): finish permissibility metric transition - #305
fix(metrics): finish permissibility metric transition#305Jake Present (jakepresent) wants to merge 2 commits into
Conversation
0e75256 to
93aec77
Compare
Chang Liu (changliu2)
left a comment
There was a problem hiding this comment.
The five gaps this fixes are real — I diffed against main and confirmed each one (the missing "Scenario permissible violations" column, results status still listing the superseded pair while the viewer had already retired it, --metric unable to even name the derived split metrics, and _log_run_headline never loading the taxonomy so it couldn't compute the split). The viewer dead-code removals are clean: no remaining reads of RunMetrics.counts / .policy_violation_rate / .overrefusal_rate, and aggregateRunViolationRate had no callers. I reproduced your test results locally — 1216 passed / 22 skipped / 474 subtests, and npm run check is 0 errors / 6 pre-existing warnings.
Two blockers, both from the same root cause: the CLI detects "split is available" by key presence rather than data presence, and _compute_prompt_metrics emits those keys whenever the suite has any behavior_categories — regardless of whether the run's node judgments resolve against them.
1. results status can render an empty dimensions table and hide real results. With a fully-judged run whose node_judgments don't match the current taxonomy (routine whenever the taxonomy is regenerated between runs in a suite), both split buckets have count == 0 but both keys are present, so _visible_dimension_summaries hides policy_violation/overrefusal and prints nothing else:
Total 1 - Scored 1 - Judge failure rate 0.0%
Dimension Summary Scored Distribution
Impermissible behavior violated - 0 0 flagged / 0 pass
Permissible behavior violated - 0 0 flagged / 0 pass
The underlying run has policy_violation_rate: 1.0 and overrefusal_rate: 1.0 — on main the user sees both. Worth noting the viewer doesn't have this bug, and permissibility.ts says why in its own docstring: dropSupersededMetrics requires "a list already narrowed to metrics carrying data." The CLI port dropped that precondition, so this adds a viewer/CLI asymmetry in a PR whose goal is to remove them. Could _visible_dimension_summaries gate on policy_violation_on_{not_,}permissible having count > 0?
2. results compare silently switches metric and drops the delta table. _resolve_compare_metric returns the split metric if any run reports the keys, so one split-capable run flips the default for the whole comparison. On the same fixture, compare run-1 run-2 renders - for run-1 and omits "Top behavior category deltas" entirely (every row returns None from _row_metric_value, so first_map is empty), where --metric policy_violation shows 100% for both runs. No warning is printed. Suggest requiring the split to carry data for every compared run before switching the default, or falling back with a one-line notice.
Two non-blocking notes:
- Base branch. This targets
ango10/assert-acs-skill-main(#303), notmain, sogh pr checksreports no checks at all —build.ymlandbuild-viewer.ymlare bothpull_request: branches: [main]. A metrics-correctness fix is currently unvalidated by CI and coupled to an ~80-commit examples PR. The diff is self-contained; could it be rebased ontomainand retargeted? permissible_overrefusal_rateis dropped fromresults status --json, which reads against "keep the legacy calculations in ... JSON output" in the description. I found no in-repo consumer (the export script computes its own), so this is probably fine — but since--jsonconsumers are the ones we can't grep for, please either restore it or call the removal out explicitly.
Separately, five agent-instruction files still tell agents the viewer renders "Harm (non-permissible)" when the real label is "Impermissible behavior violated" (.github/prompts/run-assert-eval.prompt.md:112,149, .claude/skills/run-assert-eval/SKILL.md:272,340, workflows/govern-and-remeasure.md:108, workflows/measure-clarity-failures.md:177, .cursor/rules/assert.mdc:148,186). Those come from the base branch, so #303 may be the better place — but this is the PR claiming to finish the transition.
Summary
Finish the user-facing permissibility-metric transition while preserving the legacy backend contract:
policy_violation/overrefusaldisplay for legacy runs without taxonomy--metriccomparisons for downstream compatibilityTesting
pytest tests/ -x -q→ 1217 passed, 21 skipped, 474 subtests passednpm run check --prefix viewer→ 0 errors (6 pre-existing warnings)npm run build --prefix viewerpython -m compileall -q assert_ai tests/test_results.py tests/test_runner_headline.pygit diff --checkCompatibility
This does not remove the
policy_violationoroverrefusaljudge dimensions or their artifact/JSON fields. Analysis, benchmark, and standalone export semantics remain unchanged pending a separate decision on whether they should report both split measures or retain the combined metric.