Problem
Closed-book benchmark runs (no network egress at task time) cannot use the Hermes agent, because
the Harbor task image installs it at runtime via curl | bash. With egress blocked the install
fails and the task cannot run.
Proposal
Pre-bake the NousResearch hermes-agent into the dataset image at build time, alongside the agents
already pre-baked there (Claude Code, Codex, OpenCode, Node). Pin the version through the existing
benchmark/agent-versions.env mechanism so runs stay reproducible, and add an "already installed"
guard so the runtime installer short-circuits instead of failing.
Scope
benchmark/ only — no library or routing code. Roughly 130 lines across three files.
Notes
This is benchmark tooling and stays Python/shell; it is unaffected by the ongoing Rust migration.
Happy to open the PR — the change is already written and tested against current main.
Problem
Closed-book benchmark runs (no network egress at task time) cannot use the Hermes agent, because
the Harbor task image installs it at runtime via
curl | bash. With egress blocked the installfails and the task cannot run.
Proposal
Pre-bake the NousResearch
hermes-agentinto the dataset image at build time, alongside the agentsalready pre-baked there (Claude Code, Codex, OpenCode, Node). Pin the version through the existing
benchmark/agent-versions.envmechanism so runs stay reproducible, and add an "already installed"guard so the runtime installer short-circuits instead of failing.
Scope
benchmark/only — no library or routing code. Roughly 130 lines across three files.Notes
This is benchmark tooling and stays Python/shell; it is unaffected by the ongoing Rust migration.
Happy to open the PR — the change is already written and tested against current
main.