fix(native-eval): make OpenClaw Tool Search controls explicit

[vincentkoc]

Summary

ShellBench's OpenClaw native harness does not currently encode the disabled
Tool Search control explicitly. When a run omits openclaw_tool_search_mode,
the generated OpenClaw config omits tools.toolSearch, so the installed
OpenClaw default decides the mode.

Impact

An intended Tool Search "off" ablation can silently execute with code mode
enabled. That contaminates score, token, tool-call, and latency comparisons
because the control and treatment are not actually distinct.

This occurred in an exact candidate run on July 29, 2026: the run index had no
mode, but the exported OpenClaw trace reported codeModeEngaged: true.

Expected behavior

• No mode means the harness writes tools.toolSearch: false.
• directory and code are explicit supported values.
• The selected value survives fleet dispatch, manifests, and automatic reruns.
• Empty fleet environment values normalize to unset metadata, not "".

Verification

Add focused runner and fleet tests covering explicit off, code mode, dispatch,
manifest normalization, and rerun preservation.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed September 4, 2026, 3:53 AM ET / 07:53 UTC.

Summary
Current main still lacks the requested native Tool Search controls. The member-authored issue remains necessary, and #62 is its open, same-author implementation candidate.

Reproducibility: yes. for the reported omission: native config generation has no Tool Search setting, and the run model has no mode field. The July 29 runtime outcome was not replayed.

Next step
Review and land or narrow #62; opening another repair PR would duplicate the paired implementation.

Review details

Best possible solution:

Give every native OpenClaw run a version-bound, explicit tool mode that survives dispatch and reruns, with observed mode evidence kept alongside results.

Do we have a high-confidence way to reproduce the issue?

Yes for the reported omission: native config generation has no Tool Search setting, and the run model has no mode field. The July 29 runtime outcome was not replayed.

Is this the best way to solve the issue?

Yes to explicit native controls: the existing container toggle does not cover this runner. The member’s request and paired PR already specify the intended direction; the broader PR needs its own implementation review.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• Historical runs without effective-mode provenance cannot safely be assumed comparable with a corrected explicit-off baseline.
• The July 29 runtime engagement is reported evidence; this review independently verifies the configuration omission, not that experiment’s effective tool surface.

Codex review notes: model internal, reasoning high; reviewed against c1a79f731541.

Label changes

Label justifications:

• P2: Uncontrolled benchmark modes undermine research comparisons, with impact confined to evaluation campaigns.
• impact:other: The affected outcome is benchmark validity across scores, token counts, tool calls, and latency.

Evidence reviewed

What I checked:

• Verified checkout and policy scope: Origin identifies openclaw/shellbench; HEAD and origin/main match the supplied revision. No AGENTS.md, VISION.md, maturity scorecard, or taxonomy file was found. The checkout remained clean. (c1a79f731541)
• Native configuration omission: The OpenClaw builder generates tools with only a message deny list, then writes that configuration before and after setup; it never sets toolSearch. Runtime.run_trial consumes this builder directly, establishing the affected execution path. (scripts/native_eval/harnesses.py:584, c1a79f731541)
• Mode selection is absent from native run identity: RunSpec has reasoning effort but no Tool Search mode. Planning serializes this structure, while fleet dispatch and rerun reconstruction preserve other named controls without a Tool Search field. (scripts/native_eval/models.py:24, c1a79f731541)
• Existing toggle belongs to another runner: The container adapter supports CLAWBENCH_OPENCLAW_TOOL_SEARCH, but native execution instead calls build_harness_command and runs its generated setup. That existing boolean toggle does not provide the requested native fleet control. (scripts/container_adapter_eval.sh:509, c1a79f731541)
• Existing tests establish the extension points: Runner tests inspect generated OpenClaw setup, and fleet tests check preserved reasoning effort on reruns. A tracked-tree search found no Tool Search or Code Mode coverage in the native runner, its focused tests, or docs; these tests were inspected, not executed. (tests/test_native_eval_fleet.py:1275, c1a79f731541)
• Paired implementation and reported experiment: The supplied GitHub snapshot formally links the open fix(native-eval): benchmark genuine OpenClaw code mode #62 as a closing candidate. Its body distinguishes genuine Code Mode from the legacy bridge and leaves a fresh matched benchmark batch unchecked. feat(native-eval): support OpenClaw candidate packages #57 independently records the reported control contamination while separating it from package-staging validation.

Likely related people:

• vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.