Skip to content

business_letters scorer crashes on non-conforming model responses #111

Description

@MHindermann

Summary

Scoring business_letters for command-a-vision-07-2025 (T1142) crashes with AttributeError: 'NoneType' object has no attribute 'send_date' when the model's response doesn't conform to the required structured output.

Repro

Run T1142 (scripts/run_benchmarks_command-a-vision-07-2025.py). Cohere returns the four fields at top level plus an extra content key instead of the expected metadata-wrapped structure.

Root cause (benchmarks/business_letters/benchmark.py)

  1. Cohere structured-output validation fails (metadata Field required) → falls back to JSON mode → answer.parsed = {letter_title, send_date, sender_persons, receiver_persons,
    content} (no metadata wrapper).
  2. score_request_answer: data["metadata"] → KeyError → fallback _initialize_letter(raw_letter=data).
  3. _initialize_letter calls Letter(**raw_letter); Letter has no content field → TypeError → returns None.
  4. _score_send_date(predicted_letter=None) → None.send_date → crash.

Expected behavior

A model that fails to return the required structured output should be scored as a failure for that item (0 TP, all ground-truth as FN), not crash the run, and not be salvaged by stripping non-conforming fields, which would credit a model for ignoring the schema.

Proposed fix

  • score_request_answer: when response_letter is None (parsing failed for any reason), return the complete-failure score (0 TP, all FN)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Bug.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions