Summary
Scoring business_letters for command-a-vision-07-2025 (T1142) crashes with AttributeError: 'NoneType' object has no attribute 'send_date' when the model's response doesn't conform to the required structured output.
Repro
Run T1142 (scripts/run_benchmarks_command-a-vision-07-2025.py). Cohere returns the four fields at top level plus an extra content key instead of the expected metadata-wrapped structure.
Root cause (benchmarks/business_letters/benchmark.py)
- Cohere structured-output validation fails (metadata Field required) → falls back to JSON mode → answer.parsed = {letter_title, send_date, sender_persons, receiver_persons,
content} (no metadata wrapper).
- score_request_answer: data["metadata"] → KeyError → fallback _initialize_letter(raw_letter=data).
- _initialize_letter calls Letter(**raw_letter); Letter has no content field → TypeError → returns None.
- _score_send_date(predicted_letter=None) → None.send_date → crash.
Expected behavior
A model that fails to return the required structured output should be scored as a failure for that item (0 TP, all ground-truth as FN), not crash the run, and not be salvaged by stripping non-conforming fields, which would credit a model for ignoring the schema.
Proposed fix
- score_request_answer: when response_letter is None (parsing failed for any reason), return the complete-failure score (0 TP, all FN)
Summary
Scoring business_letters for command-a-vision-07-2025 (T1142) crashes with AttributeError: 'NoneType' object has no attribute 'send_date' when the model's response doesn't conform to the required structured output.
Repro
Run T1142 (scripts/run_benchmarks_command-a-vision-07-2025.py). Cohere returns the four fields at top level plus an extra content key instead of the expected metadata-wrapped structure.
Root cause (benchmarks/business_letters/benchmark.py)
content} (no metadata wrapper).
Expected behavior
A model that fails to return the required structured output should be scored as a failure for that item (0 TP, all ground-truth as FN), not crash the run, and not be salvaged by stripping non-conforming fields, which would credit a model for ignoring the schema.
Proposed fix