feat(hle): add hle_250 — fixed 250-question subset, 10 epochs, OpenRouter judge default - #3
Merged
Merged
Conversation
|
✅ Benchmark documentation has been automatically updated. |
… judge default Mirrors the shape of Artificial Analysis's Endpoint Accuracy Index HLE-250 (their exact subset is private): 250 text-only questions stratified proportionally over the 8 HLE categories (seed 20260807, largest-remainder rounding), committed as data in hle_250_ids.py so the subset can never drift. Runs 10 epochs averaged at temperature=1.0/top_p=0.95, per the reasoning-eval sampling convention, and defaults the grader to GPT-5.6 Luna (medium) via OpenRouter (OPENROUTER_API_KEY). get_dataset grows ids= and name= filters that refuse silently-missing ids; hle_scorer grows a reasoning_effort argument for graders that support one. Verified against the gated cais/hle dataset: 250 unique text-only samples, epochs=10, and one real judge call through inspect's openrouter provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nhat-di
changed the base branch from
deepinfra-service-tier-053
to
deepinfra-provider
August 7, 2026 22:51
# Conflicts: # src/openbench/model/_providers/deepinfra.py # tests/test_deepinfra_provider.py
|
✅ Benchmark documentation has been automatically updated. |
…in PR diff Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nhat-di
marked this pull request as ready for review
August 12, 2026 08:51
nhat-di
added a commit
that referenced
this pull request
Aug 12, 2026
…uter judge default (#6) * feat(deepinfra): optional service_tier and an explicit request timeout Two changes to the DeepInfra provider, both needed to benchmark models on DeepInfra's own infrastructure. **service_tier via DEEPINFRA_SERVICE_TIER.** DeepInfra accepts an optional service_tier on chat completions; "flex" rides spare capacity, which is what lets a benchmark harness run without competing with production traffic. Read from the environment so a harness can set it per-run without touching eval code, and an explicit extra_body setting still wins. **An explicit per-request timeout.** The OpenAI SDK defaults to read=600s and nothing upstream raises it: openai_compatible.py builds AsyncOpenAI without a timeout, and inspect spends GenerateConfig.timeout on tenacity's stop_after_delay, which bounds retrying and never reaches the socket. So a generation legitimately running past ten minutes is cut off client-side. That was measured, not theoretical: benchmarking a 9B model on mbpp (which leaves max_tokens unset), 19-24% of requests died at exactly 599-602s while the API had admitted every one of them in about a second and refused none. Successful requests in the same window tailed to 479s, so the cut lands inside the live part of the latency distribution -- and it removes the *slowest* samples, biasing the score rather than merely shrinking it. A later attempt to set 1200s instead still produced deaths at 1199s, because the server's own budget can exceed that before generation even starts. An hour is a backstop rather than a target; a run that needs it has a problem worth seeing. Deliberately not taken from config.timeout, which inspect already spends as the retry budget -- reusing one number for both lets a single attempt consume the budget and leaves nothing to retry, which is the original failure. The value is an httpx.Timeout rather than a bare float so only the wait for tokens gets the long budget: connect stays at 30s, or an unreachable endpoint would hang for the full hour instead of failing fast. Also pins inspect-ai to 0.3.142; 0.3.141 was never published to PyPI, so the declared dependency does not install. Tests cover the timeout default, the short connect phase, independence from config.timeout, an explicit override, and all three service-tier paths. * feat(hle): add hle_250 — fixed 250-question subset, 10 epochs, OpenRouter judge default (#3) * feat(hle): add hle_250, a fixed 250-question subset with epochs and a judge default Mirrors the shape of Artificial Analysis's Endpoint Accuracy Index HLE-250 (their exact subset is private): 250 text-only questions stratified proportionally over the 8 HLE categories (seed 20260807, largest-remainder rounding), committed as data in hle_250_ids.py so the subset can never drift. Runs 10 epochs averaged at temperature=1.0/top_p=0.95, per the reasoning-eval sampling convention, and defaults the grader to GPT-5.6 Luna (medium) via OpenRouter (OPENROUTER_API_KEY). get_dataset grows ids= and name= filters that refuse silently-missing ids; hle_scorer grows a reasoning_effort argument for graders that support one. Verified against the gated cais/hle dataset: 250 unique text-only samples, epochs=10, and one real judge call through inspect's openrouter provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update benchmark docs [skip ci] --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
hle_250, QualityMonitor's AA-shaped HLE benchmark (see deepinfra/backend quality-monitor plan).What
hle_250task: fixed 250-question text-only HLE subset,Epochs(10, "mean"),temperature=1.0/top_p=0.95, grader defaults toopenrouter/openai/gpt-5.6-lunawithreasoning_effort=medium(needsOPENROUTER_API_KEY).hle_250_ids.py: the subset as committed data — proportional stratification over the 8 HLE categories (2,158 text-only questions, May 2025 revision), seed 20260807, largest-remainder rounding. Committed as ids, not a seed, so dataset reordering can never move the subset.get_dataset(ids=, name=): subset filter that raises on missing ids rather than silently shrinking.hle_scorer(reasoning_effort=): grader effort pass-through.Why not AA's actual HLE-250
Artificial Analysis's subset is private and difficulty-overweighted; this is an equivalent-shape instrument (same size, repeats, sampling convention, judge family), not a replica. Scores are not comparable to AA's published numbers — that is documented on the task.
Base
Rebased onto
deepinfra-provider@aa2eef8, the branch the deployedopenbench:service-tier-aa2eef860-r1image is built from — the diff is exactly the hle_250 delta. (An earlier revision of this PR was based on the pre-consolidationdeepinfra-service-tier-053chain; that branch was restored at83ff858during the archaeology and can be deleted again if unwanted.)Validation
cais/hledataset: 250 unique text-only samples, epochs=10, config 1.0/0.95/8192.get_all_benchmarks()(top-level, not subtask).openrouter/provider with the prod key: correct verdict format returned.🤖 Generated with Claude Code