Fix evaluator-accuracy and confusion-matrix misalignment when a rating fails to parse

[arthi-arumugam-git]

What

calc_metrics scores the evaluator's predicted persuasion degrees against the intended ones, per user. Both the accuracy metrics (evaluator_accuracy_metrics.json) and the confusion matrix (all_metrics.json) locate a user's prediction by their position within their conspiracy group, but they read that position out of filtered_ratings_by_turn, which has had non-numeric ratings removed.

extract_rating returns the string "Format error" whenever the judge's output cannot be parsed as ... | N. Filtering that out shrinks the group's rating list and shifts every later user's position by one, so:

• predictions are scored against the wrong users' intended degrees, and
• the trailing user(s) in each affected group get position >= len(...) and are silently dropped.

The result is wrong overall_accuracy_by_turn, mean_absolute_error_by_turn, mean_squared_error_by_turn, degree_specific_accuracy, and confusion matrices, with nothing raised.

Reproduction

Three users share one conspiracy with true degrees [0, 1, 2]. The evaluator predicts every parseable one correctly, but user 0's rating is a "Format error":

	accuracy	MAE
before	0.0	1.0
correct	1.0	0.0

A perfectly accurate evaluator is reported as completely inaccurate: after filtering, user 0 (true 0) reads [1, 2][0] = 1, user 1 (true 1) reads [1, 2][1] = 2, and user 2 falls off the end. Both remaining users are scored wrong, so accuracy is 0/2 and MAE is 2/2.

Fix

Add get_aligned_prediction(), which reads the raw (unfiltered) ratings_by_turn so a user's position indexes their own prediction, and returns None for a missing or non-numeric rating so callers skip it in place instead of compacting the list. Both the accuracy loop and the confusion-matrix loop use it. Per-title averages and distributions keep using filtered_ratings_by_turn, since those are order-independent.

The plotted confusion matrix had the same bug

The confusion matrix written to evaluator_accuracy_metrics.json and the one rendered to persuasion_degree_confusion_matrix.png are built by two separate pieces of code. Fixing only the metrics one would have left the PNG misaligned and made the two artifacts disagree with each other, so this also updates create_visualizations.

That block indexed filtered_ratings_by_turn by each user's position within their conspiracy group, exactly as the metrics code did, with the same consequences. It now calls the same get_aligned_prediction helper.

This resolves the pre-existing # todo: fix off by one error here comment on that block, which is removed. The turn index was already correct and is unchanged: ratings_by_turn and filtered_ratings_by_turn always have one entry per turn, so last_turn_idx still selects the final turn and the "(Final Turn)" title stays accurate.

Tests

tests/test_calc_metrics.py covers the format-error case at both the helper and full-calc_metrics level, plus an all-numeric control.

tests/test_visualizations.py is new and covers the rendered plot rather than the metrics path: it monkeypatches Axes.imshow and asserts on the array actually handed to the plot.

• test_plotted_confusion_matrix_skips_format_errors_without_shifting
• test_plotted_confusion_matrix_all_numeric_unchanged

Both fail without the visualization change, with the mass landing off the diagonal. Five tests pass together:

tests/test_calc_metrics.py ...                                           [ 60%]
tests/test_visualizations.py ..                                          [100%]
============================== 5 passed in 3.03s ==============================

black, isort (--profile black), and flake8 (--ignore=E203,E704,E501,W503) are clean, run with the args from .pre-commit-config.yaml. mypy reports no issues on the touched files under --python-version 3.13. Under the repo's configured python_version = 3.11 mypy fails inside numpy's own stubs (numpy/__init__.pyi:737: Type statement is only supported in Python 3.12 and greater); that reproduces on files this PR does not touch, so it looks like a local numpy version issue rather than anything here. tests/test_main.py was not run, since it imports generate_conversations and pulls in torch.

Related sites, deliberately not in this PR

While tracing this I found the same root cause in about ten other places, and I would rather name them than quietly widen the diff:

• src/utils/utils.py, print_results_to_terminal: zips filtered ratings positionally against refusals_by_turn, which is built in raw user order, so refusal flags get attributed to the wrong users and the last user's refusal is never checked. This is a ratings-vs-refusals pairing rather than ratings-vs-users, and the loop has no user_idx, so get_aligned_prediction does not drop in cleanly.
• Roughly eight further sites in visualizations.py, plus main.py (the logviz per-sample rating) and generate_persuasion_degree_plots.py, use the same positional pattern. Most are the refusals variant above.

Fixing all of them turns a focused bugfix into a repo-wide refactor that is hard to review. Happy to open a follow-up issue, or to fold them in here if you would prefer one pass.