fix(benchmarks): make rlpr/ifeval/bfcl_simple load() callable instead of raising TypeError (#152)#153
Closed
joaovictor91123 wants to merge 1 commit into
Conversation
… of raising TypeError (mini-router#152) Each of the three OOD benchmark facades passed ("<name>", split) positionally to its sibling load_tasks(split, *, ...), which accepts one positional -- the benchmark name belongs to the imported _load_tasks(benchmark, split, ...). Every call to the documented load() entrypoint raised: TypeError: load_tasks() takes 1 positional argument but 2 positional arguments (and 2 keyword-only arguments) were given This is the identical defect fixed for livecodebench in mini-router#86/mini-router#90; the three newer facades were copied from the pre-fix template and regressed the same way. configs/benchmarks.yaml names all three as `loader:` and benchmarks/__init__.py documents load(split, **kw) as the contract, so the public entrypoint was dead. The facade tests stayed green because they monkeypatched the sibling load_tasks (the function whose signature is the bug) with a four-positional stub, so the miscall type-checked against the fake. Patch the real _load_tasks boundary in each and add a load_is_callable regression test, mirroring the mini-router#90 fix.
0021791 to
034f42d
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #152
Summary
The three OOD benchmark facades added in #104/#103/#102 —
benchmarks/rlpr.py,benchmarks/ifeval.py,benchmarks/bfcl_simple.py— each raisedTypeErroron every call to their publicload(). This is the identical defect thatbenchmarks/livecodebench.pyhad and that was fixed + regression-tested in #90; the three newer facades were copied from the pre-fix template and regressed the same way.load()passes two positionals, but the siblingload_tasks(split, *, ...)takes one. Offline, before any dataset loading:Same for
ifevalandbfcl_simple.livecodebench.load(...)(fixed in #90) does not raise.Why This Matters
load()is the documented config-facing entrypoint.benchmarks/__init__.pystates "Each loader exposesload(split, **kw) -> list[Task]", andconfigs/benchmarks.yamlnames all three modules asloader:for the OOD suite (ifeval,rlpr,bfcl_simple). Any caller going through the documented facade — rather than reaching past it todataset.load_tasksdirectly — cannot load these three benchmarks at all. This is the same public-API break #90 fixed for LiveCodeBench, now closed for its three siblings.Why CI stayed green
Each facade test masks the bug exactly as the pre-#90 livecodebench test did — it monkeypatches the sibling
load_tasks(the very function whose signature is the bug) with a four-positional stub, so the two-positional miscall type-checks against the fake:A stub whose signature does not match the function it replaces cannot catch a signature bug. #90 documented this and fixed livecodebench's test, but the fix was never propagated to the three siblings that share the template.
What Changed
benchmarks/rlpr.py,benchmarks/ifeval.py,benchmarks/bfcl_simple.py:load()now delegates toload_tasks(split, max_items=..., seed=...), naming the benchmark in one place (matching the merged fix(benchmarks): make livecodebench.load() callable instead of raising TypeError (#86) #90 fix).tests/test_rlpr.py,tests/test_ifeval.py,tests/test_bfcl.py: each*_facade_delegatestest now patches the real_load_tasksboundary, and each gains a*_load_is_callabletest that callsload()without patchingload_tasks— the regression test fix(benchmarks): make livecodebench.load() callable instead of raising TypeError (#86) #90 established as the standard.docs/JOURNAL.md: replication-log entry per repository protocol (AGENTS.md §6).Notes
mainwith the reportedTypeError(including the corrected*_facade_delegates, which passed before only because of the mismatched stub) and pass after.pytest tests -q→ 236 passed.load()callable.