Skip to content

fix(metrics): finish permissibility metric transition - #305

Open
Jake Present (jakepresent) wants to merge 2 commits into
ango10/assert-acs-skill-mainfrom
jake/cli-permissibility-followup
Open

fix(metrics): finish permissibility metric transition#305
Jake Present (jakepresent) wants to merge 2 commits into
ango10/assert-acs-skill-mainfrom
jake/cli-permissibility-followup

Conversation

@jakepresent

@jakepresent Jake Present (jakepresent) commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Finish the user-facing permissibility-metric transition while preserving the legacy backend contract:

  • show both permissible and impermissible violation rates for prompt and scenario runs
  • make run detail, within-suite compare, cross-suite compare, and the post-run terminal headline use the split when taxonomy is available
  • retain the original policy_violation / overrefusal display for legacy runs without taxonomy
  • keep the legacy calculations in score rows, artifacts, JSON output, and explicit --metric comparisons for downstream compatibility
  • remove redundant viewer projections, combined-policy counts, and special-case branches that were left dead after feat(viewer): headline policy violations split by behavior #295
  • update the CLI reference for the taxonomy-aware compare default

Testing

  • pytest tests/ -x -q → 1217 passed, 21 skipped, 474 subtests passed
  • npm run check --prefix viewer → 0 errors (6 pre-existing warnings)
  • npm run build --prefix viewer
  • focused headline/results/viewer tests → 39 passed
  • python -m compileall -q assert_ai tests/test_results.py tests/test_runner_headline.py
  • git diff --check

Compatibility

This does not remove the policy_violation or overrefusal judge dimensions or their artifact/JSON fields. Analysis, benchmark, and standalone export semantics remain unchanged pending a separate decision on whether they should report both split measures or retain the combined metric.

@jakepresent
Jake Present (jakepresent) force-pushed the jake/cli-permissibility-followup branch from 0e75256 to 93aec77 Compare August 6, 2026 21:55
@jakepresent Jake Present (jakepresent) changed the title fix(cli): finish permissibility metric alignment fix(metrics): finish permissibility metric transition Aug 6, 2026

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The five gaps this fixes are real — I diffed against main and confirmed each one (the missing "Scenario permissible violations" column, results status still listing the superseded pair while the viewer had already retired it, --metric unable to even name the derived split metrics, and _log_run_headline never loading the taxonomy so it couldn't compute the split). The viewer dead-code removals are clean: no remaining reads of RunMetrics.counts / .policy_violation_rate / .overrefusal_rate, and aggregateRunViolationRate had no callers. I reproduced your test results locally — 1216 passed / 22 skipped / 474 subtests, and npm run check is 0 errors / 6 pre-existing warnings.

Two blockers, both from the same root cause: the CLI detects "split is available" by key presence rather than data presence, and _compute_prompt_metrics emits those keys whenever the suite has any behavior_categories — regardless of whether the run's node judgments resolve against them.

1. results status can render an empty dimensions table and hide real results. With a fully-judged run whose node_judgments don't match the current taxonomy (routine whenever the taxonomy is regenerated between runs in a suite), both split buckets have count == 0 but both keys are present, so _visible_dimension_summaries hides policy_violation/overrefusal and prints nothing else:

Total 1 - Scored 1 - Judge failure rate 0.0%
Dimension                        Summary  Scored  Distribution
Impermissible behavior violated  -        0       0 flagged / 0 pass
Permissible behavior violated    -        0       0 flagged / 0 pass

The underlying run has policy_violation_rate: 1.0 and overrefusal_rate: 1.0 — on main the user sees both. Worth noting the viewer doesn't have this bug, and permissibility.ts says why in its own docstring: dropSupersededMetrics requires "a list already narrowed to metrics carrying data." The CLI port dropped that precondition, so this adds a viewer/CLI asymmetry in a PR whose goal is to remove them. Could _visible_dimension_summaries gate on policy_violation_on_{not_,}permissible having count > 0?

2. results compare silently switches metric and drops the delta table. _resolve_compare_metric returns the split metric if any run reports the keys, so one split-capable run flips the default for the whole comparison. On the same fixture, compare run-1 run-2 renders - for run-1 and omits "Top behavior category deltas" entirely (every row returns None from _row_metric_value, so first_map is empty), where --metric policy_violation shows 100% for both runs. No warning is printed. Suggest requiring the split to carry data for every compared run before switching the default, or falling back with a one-line notice.

Two non-blocking notes:

  • Base branch. This targets ango10/assert-acs-skill-main (#303), not main, so gh pr checks reports no checks at all — build.yml and build-viewer.yml are both pull_request: branches: [main]. A metrics-correctness fix is currently unvalidated by CI and coupled to an ~80-commit examples PR. The diff is self-contained; could it be rebased onto main and retargeted?
  • permissible_overrefusal_rate is dropped from results status --json, which reads against "keep the legacy calculations in ... JSON output" in the description. I found no in-repo consumer (the export script computes its own), so this is probably fine — but since --json consumers are the ones we can't grep for, please either restore it or call the removal out explicitly.

Separately, five agent-instruction files still tell agents the viewer renders "Harm (non-permissible)" when the real label is "Impermissible behavior violated" (.github/prompts/run-assert-eval.prompt.md:112,149, .claude/skills/run-assert-eval/SKILL.md:272,340, workflows/govern-and-remeasure.md:108, workflows/measure-clarity-failures.md:177, .cursor/rules/assert.mdc:148,186). Those come from the base branch, so #303 may be the better place — but this is the PR claiming to finish the transition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants