I’ve published a conformance census of the default judge templates across nine eval frameworks: https://github.com/idilgozel/evaluator-integrity
I found two things in autoevals that I think are worth flagging.
closed_q_a.yaml drops a scoping clause.OpenAI’s original template says the reasoning instruction applies to “your reasoning about the criterion.” The autoevals version drops that part. The resulting COT_SUFFIX therefore gives the model an unscoped instruction to reason step by step, and this is appended in code.
Record: conformance/corpus/braintrust-autoevals__closedqa.json
factuality.yaml has “in a step-by-step matter”The original OpenAI wording is “in a step by step manner.” The autoevals version has “in a step-by-step matter.”*This looks like a small mutation of the same template, so it may also be useful as a lineage fingerprint when comparing where these templates came from.
There is also a scoring issue I think is worth looking at. E (“differences don’t matter”) gets 1.0 without defining what counts as a difference that doesn’t matter. Also, B (correct answer + unverified extra claims) gets 0.6, while A (correct but incomplete) gets 0.4. So the default scoring gives a higher score to an answer containing unchecked additional claims than to one that is simply incomplete.
Record: conformance/corpus/braintrust-autoevals__factuality.json
I’m happy to open a PR for the scoping clause and typo if useful. I didn’t want to open one without checking first.
I’ve published a conformance census of the default judge templates across nine eval frameworks: https://github.com/idilgozel/evaluator-integrity
I found two things in
autoevalsthat I think are worth flagging.closed_q_a.yamldrops a scoping clause.OpenAI’s original template says the reasoning instruction applies to “your reasoning about the criterion.” Theautoevalsversion drops that part. The resulting COT_SUFFIX therefore gives the model an unscoped instruction to reason step by step, and this is appended in code.Record:
conformance/corpus/braintrust-autoevals__closedqa.jsonfactuality.yamlhas “in a step-by-step matter”The original OpenAI wording is “in a step by step manner.” Theautoevalsversion has “in a step-by-step matter.”*This looks like a small mutation of the same template, so it may also be useful as a lineage fingerprint when comparing where these templates came from.There is also a scoring issue I think is worth looking at. E (“differences don’t matter”) gets 1.0 without defining what counts as a difference that doesn’t matter. Also, B (correct answer + unverified extra claims) gets 0.6, while A (correct but incomplete) gets 0.4. So the default scoring gives a higher score to an answer containing unchecked additional claims than to one that is simply incomplete.
Record:
conformance/corpus/braintrust-autoevals__factuality.jsonI’m happy to open a PR for the scoping clause and typo if useful. I didn’t want to open one without checking first.