feat(cli): results matrix — behavior × arm comparison table - #274
Open
Chang Liu (changliu2) wants to merge 3 commits into
Open
feat(cli): results matrix — behavior × arm comparison table#274Chang Liu (changliu2) wants to merge 3 commits into
Chang Liu (changliu2) wants to merge 3 commits into
Conversation
Adds `assert-ai results matrix` rendering a 2D pivot (rows=behavior, cols=arm/variant, cells=metric rate) over multiple runs, for the N-arms × M-behaviors comparison used by the 1-behavior-per-yaml pattern. Pivots on each run's config.yaml behavior.name and an arm label derived from the run id. Includes --suite auto-expand, --json, and tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3
Chang Liu (changliu2)
requested review from
Aaron Aspinwall (AaronAspinwall123),
Jake Present (jakepresent) and
tangym
as code owners
July 22, 2026 21:54
`policy_violation` unions permissible and impermissible behaviors, so ranking behaviors by it can order them by the wrong thing entirely. On real runs the two halves diverge sharply -- a behavior can carry a high union rate made up almost wholly of mishandled *permissible* work while another with a lower union rate is nearly all genuine impermissible failure. A behavior x arm matrix is precisely the surface where that ordering matters, since its whole purpose is to say which behavior is worst. So the matrix now defaults to the impermissible half whenever every run reports the split, matching the supersede rule `results list` and `results status` already follow. It requires *all* runs to have it rather than any: one run contributing an impermissible-only rate while another contributes the union would put non-comparable numbers in the same table, which is worse than falling back to the union everywhere. Runs without a taxonomy -- including quality suites that repurpose `policy_violation` for non-safety failures -- keep reporting the union. Fixes a bug in the process. The split is derived from node judgments plus the taxonomy and is stored as a top-level rate, not under `dimensions`, so `_run_dimension_rate` could not see it. Passing `--metric policy_violation_not_permissible` resolved and *labelled* correctly and then rendered every cell as `-`, which reads as "no violations" rather than "not wired up". Both spellings are now accepted: the viewer-facing metric name and the artifact rate key. Also notes the denominators. Each half is scored only over the rows where a behavior in that bucket was relevant, so the halves differ from each other and from `policy_violation` -- on the career-health CV-injection baseline the impermissible half is 4/21 while the permissible half is 16/25. Without saying so, a reader will try to add them and find they do not reconcile to the union. Six tests, each verified to fail against the previous behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
assert-ai results matrix, a 2D comparison view for multi-run eval results. Companion to the bank-manager one-behavior-per-yaml configs in #262 — it renders the full N-failure-modes × M-arms grid that the viewer's single-suite compare view can't.What changed
assert-ai results matrix SUITE/RUN ...under theresultsgroup.--suite <SUITE>auto-expands all scored runs in a suite; pass several to span multiple behavior-suites in one table.--metric(defaultpolicy_violation),--json,--no-color,--results-dir.config.yamlbehavior.name(falls back to manifest, then suite id).baseline → prompted → acsthen any others alphabetically (deterministic).-in tables andnullin JSON.Example
Demo output (rendered from real runs)
Validation