Skip to content

feat(cli): results matrix — behavior × arm comparison table - #274

Open
Chang Liu (changliu2) wants to merge 3 commits into
mainfrom
changliu2/cli-results-matrix
Open

feat(cli): results matrix — behavior × arm comparison table#274
Chang Liu (changliu2) wants to merge 3 commits into
mainfrom
changliu2/cli-results-matrix

Conversation

@changliu2

Copy link
Copy Markdown
Collaborator

Adds assert-ai results matrix, a 2D comparison view for multi-run eval results. Companion to the bank-manager one-behavior-per-yaml configs in #262 — it renders the full N-failure-modes × M-arms grid that the viewer's single-suite compare view can't.

What changed

  • New assert-ai results matrix SUITE/RUN ... under the results group.
  • Repeatable --suite <SUITE> auto-expands all scored runs in a suite; pass several to span multiple behavior-suites in one table.
  • --metric (default policy_violation), --json, --no-color, --results-dir.
  • Rows = behaviors, read from each run's config.yaml behavior.name (falls back to manifest, then suite id).
  • Columns = arm labels derived from run ids, ordered baseline → prompted → acs then any others alphabetically (deterministic).
  • Cells = the metric rate; missing behavior/arm cells render - in tables and null in JSON.

Example

assert-ai results matrix `
  --suite bank-1b-distortion --suite bank-1b-data-leak `
  --suite bank-1b-policy-fab --suite bank-1b-unauth-txn `
  --metric policy_violation

Demo output (rendered from real runs)

       Behavior × arm matrix (Policy violation)
Behavior                      baseline  prompted  acs
no_financial_distortion       96.0%     88.0%     89.8%
no_sensitive_data_leak        53.3%     57.8%     45.5%
no_fabricated_policy          92.0%     84.0%     87.8%
no_unauthorized_transactions  60.0%     50.0%     26.0%
      Behavior × arm matrix (Strict overrefusal)
Behavior                      baseline  prompted  acs
no_financial_distortion       2.0%      0.0%      8.2%
no_sensitive_data_leak        2.2%      6.7%      2.3%
no_fabricated_policy          0.0%      0.0%      0.0%
no_unauthorized_transactions  2.0%      0.0%      2.0%

Validation

8 passed, 4 skipped, 1163 deselected

Adds `assert-ai results matrix` rendering a 2D pivot (rows=behavior,
cols=arm/variant, cells=metric rate) over multiple runs, for the
N-arms × M-behaviors comparison used by the 1-behavior-per-yaml pattern.
Pivots on each run's config.yaml behavior.name and an arm label derived
from the run id. Includes --suite auto-expand, --json, and tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3
`policy_violation` unions permissible and impermissible behaviors, so ranking
behaviors by it can order them by the wrong thing entirely. On real runs the two
halves diverge sharply -- a behavior can carry a high union rate made up almost
wholly of mishandled *permissible* work while another with a lower union rate is
nearly all genuine impermissible failure. A behavior x arm matrix is precisely
the surface where that ordering matters, since its whole purpose is to say which
behavior is worst.

So the matrix now defaults to the impermissible half whenever every run reports
the split, matching the supersede rule `results list` and `results status`
already follow. It requires *all* runs to have it rather than any: one run
contributing an impermissible-only rate while another contributes the union would
put non-comparable numbers in the same table, which is worse than falling back to
the union everywhere. Runs without a taxonomy -- including quality suites that
repurpose `policy_violation` for non-safety failures -- keep reporting the union.

Fixes a bug in the process. The split is derived from node judgments plus the
taxonomy and is stored as a top-level rate, not under `dimensions`, so
`_run_dimension_rate` could not see it. Passing
`--metric policy_violation_not_permissible` resolved and *labelled* correctly and
then rendered every cell as `-`, which reads as "no violations" rather than "not
wired up". Both spellings are now accepted: the viewer-facing metric name and the
artifact rate key.

Also notes the denominators. Each half is scored only over the rows where a
behavior in that bucket was relevant, so the halves differ from each other and
from `policy_violation` -- on the career-health CV-injection baseline the
impermissible half is 4/21 while the permissible half is 16/25. Without saying
so, a reader will try to add them and find they do not reconcile to the union.

Six tests, each verified to fail against the previous behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants