Skip to content

fix(benchmarks): make rlpr/ifeval/bfcl_simple load() callable instead of raising TypeError (#152)#153

Closed
joaovictor91123 wants to merge 1 commit into
mini-router:mainfrom
joaovictor91123:sn74-joaovictor91123-benchmark-facades-typeerror
Closed

fix(benchmarks): make rlpr/ifeval/bfcl_simple load() callable instead of raising TypeError (#152)#153
joaovictor91123 wants to merge 1 commit into
mini-router:mainfrom
joaovictor91123:sn74-joaovictor91123-benchmark-facades-typeerror

Conversation

@joaovictor91123

Copy link
Copy Markdown
Contributor

Fixes #152

Summary

The three OOD benchmark facades added in #104/#103/#102benchmarks/rlpr.py, benchmarks/ifeval.py, benchmarks/bfcl_simple.py — each raised TypeError on every call to their public load(). This is the identical defect that benchmarks/livecodebench.py had and that was fixed + regression-tested in #90; the three newer facades were copied from the pre-fix template and regressed the same way.

def load(split, *, max_items=None, seed=0, allow_toy_fallback=False):
    return load_tasks(          # <-- the SIBLING load_tasks, not _load_tasks
        "rlpr", split, max_items=max_items, seed=seed,
    )

def load_tasks(split, *, max_items=None, seed=0, allow_toy_fallback=False):
    return _load_tasks("rlpr", split, max_items=max_items, seed=seed)

load() passes two positionals, but the sibling load_tasks(split, *, ...) takes one. Offline, before any dataset loading:

>>> import benchmarks.rlpr, benchmarks.ifeval, benchmarks.bfcl_simple
>>> benchmarks.rlpr.load("test", max_items=1)
TypeError: load_tasks() takes 1 positional argument but 2 positional arguments (and 2 keyword-only arguments) were given

Same for ifeval and bfcl_simple. livecodebench.load(...) (fixed in #90) does not raise.

Why This Matters

load() is the documented config-facing entrypoint. benchmarks/__init__.py states "Each loader exposes load(split, **kw) -> list[Task]", and configs/benchmarks.yaml names all three modules as loader: for the OOD suite (ifeval, rlpr, bfcl_simple). Any caller going through the documented facade — rather than reaching past it to dataset.load_tasks directly — cannot load these three benchmarks at all. This is the same public-API break #90 fixed for LiveCodeBench, now closed for its three siblings.

Why CI stayed green

Each facade test masks the bug exactly as the pre-#90 livecodebench test did — it monkeypatches the sibling load_tasks (the very function whose signature is the bug) with a four-positional stub, so the two-positional miscall type-checks against the fake:

def fake_load_tasks(benchmark, split, max_items, seed):   # four positionals
    ...
monkeypatch.setattr(RLPR, "load_tasks", fake_load_tasks)

A stub whose signature does not match the function it replaces cannot catch a signature bug. #90 documented this and fixed livecodebench's test, but the fix was never propagated to the three siblings that share the template.

What Changed

Notes

  • Verified locally: all 6 touched/new facade tests fail on pre-fix main with the reported TypeError (including the corrected *_facade_delegates, which passed before only because of the mismatched stub) and pass after. pytest tests -q → 236 passed.
  • The HF-row parsing and scoring tests in the same three files are untouched and still pass, so the loading/grading paths are unchanged — this only repairs the facade call.
  • Same one-line delegation change in all three files; no behavior change beyond making load() callable.

@github-actions github-actions Bot added benchmark Benchmark or config changes docs Documentation changes miner Miner contribution PR labels Jul 13, 2026
… of raising TypeError (mini-router#152)

Each of the three OOD benchmark facades passed ("<name>", split) positionally
to its sibling load_tasks(split, *, ...), which accepts one positional -- the
benchmark name belongs to the imported _load_tasks(benchmark, split, ...). Every
call to the documented load() entrypoint raised:

    TypeError: load_tasks() takes 1 positional argument but 2 positional
    arguments (and 2 keyword-only arguments) were given

This is the identical defect fixed for livecodebench in mini-router#86/mini-router#90; the three
newer facades were copied from the pre-fix template and regressed the same way.
configs/benchmarks.yaml names all three as `loader:` and benchmarks/__init__.py
documents load(split, **kw) as the contract, so the public entrypoint was dead.

The facade tests stayed green because they monkeypatched the sibling load_tasks
(the function whose signature is the bug) with a four-positional stub, so the
miscall type-checked against the fake. Patch the real _load_tasks boundary in
each and add a load_is_callable regression test, mirroring the mini-router#90 fix.
@joaovictor91123 joaovictor91123 force-pushed the sn74-joaovictor91123-benchmark-facades-typeerror branch from 0021791 to 034f42d Compare July 13, 2026 13:58
@joaovictor91123

Copy link
Copy Markdown
Contributor Author

Temporarily closing to keep two PRs in flight at a time; the branch sn74-joaovictor91123-benchmark-facades-typeerror is preserved and this will be re-opened once #175 or #176 lands. The fix (and issue #152) stand — benchmarks.{rlpr,ifeval,bfcl_simple}.load() still raise TypeError on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark Benchmark or config changes docs Documentation changes miner Miner contribution PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmarks: rlpr / ifeval / bfcl_simple load() all raise TypeError (same bug as #86, masked by the same test pattern)

1 participant