feat(deepinfra): optional service_tier and an explicit request timeout - #5
Merged
Conversation
Two changes to the DeepInfra provider, both needed to benchmark models on DeepInfra's own infrastructure. **service_tier via DEEPINFRA_SERVICE_TIER.** DeepInfra accepts an optional service_tier on chat completions; "flex" rides spare capacity, which is what lets a benchmark harness run without competing with production traffic. Read from the environment so a harness can set it per-run without touching eval code, and an explicit extra_body setting still wins. **An explicit per-request timeout.** The OpenAI SDK defaults to read=600s and nothing upstream raises it: openai_compatible.py builds AsyncOpenAI without a timeout, and inspect spends GenerateConfig.timeout on tenacity's stop_after_delay, which bounds retrying and never reaches the socket. So a generation legitimately running past ten minutes is cut off client-side. That was measured, not theoretical: benchmarking a 9B model on mbpp (which leaves max_tokens unset), 19-24% of requests died at exactly 599-602s while the API had admitted every one of them in about a second and refused none. Successful requests in the same window tailed to 479s, so the cut lands inside the live part of the latency distribution -- and it removes the *slowest* samples, biasing the score rather than merely shrinking it. A later attempt to set 1200s instead still produced deaths at 1199s, because the server's own budget can exceed that before generation even starts. An hour is a backstop rather than a target; a run that needs it has a problem worth seeing. Deliberately not taken from config.timeout, which inspect already spends as the retry budget -- reusing one number for both lets a single attempt consume the budget and leaves nothing to retry, which is the original failure. The value is an httpx.Timeout rather than a bare float so only the wait for tokens gets the long budget: connect stays at 30s, or an unreachable endpoint would hang for the full hour instead of failing fast. Also pins inspect-ai to 0.3.142; 0.3.141 was never published to PyPI, so the declared dependency does not install. Tests cover the timeout default, the short connect phase, independence from config.timeout, an explicit override, and all three service-tier paths.
# Conflicts: # src/openbench/model/_providers/deepinfra.py # tests/test_deepinfra_provider.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two additions to the DeepInfra provider (
src/openbench/model/_providers/deepinfra.py):DEEPINFRA_SERVICE_TIERenv var (e.g.flexto ride spare capacity) is injected intoextra_body.service_tieron each request. Env-based so a harness can control it without touching eval code; an explicitextra_bodysetting always wins.config.timeout, which inspect already spends as the retry budget — the two are different quantities.Tests
tests/test_deepinfra_provider.pycovers both: service-tier injection (set, unset, and explicit-extra_body-wins cases) and the timeout defaulting (applied by default, caller override respected). Full unit suite passes locally on the branch.Notes
This is the branch the deployed
openbench:service-tier-aa2eef860-r1image is built from. PR #3 (hle_250) is based on this branch and can be retargeted tomainonce this merges.🤖 Generated with Claude Code