Skip to content

No-assets mode: the full benchmark with zero RCT2 files, Linux images, OpenAI-compatible lane - #2

Open
tlrmchlsmth wants to merge 18 commits into
wseaton:evalfrom
tlrmchlsmth:linux-support
Open

No-assets mode: the full benchmark with zero RCT2 files, Linux images, OpenAI-compatible lane#2
tlrmchlsmth wants to merge 18 commits into
wseaton:evalfrom
tlrmchlsmth:linux-support

Conversation

@tlrmchlsmth

@tlrmchlsmth tlrmchlsmth commented Jul 25, 2026

Copy link
Copy Markdown

What

eval --no-graphics runs the entire scoring path — park load, placement, entrance auto-placement, test circuits, ratings, the drawability gate — with zero RollerCoaster Tycoon 2 files. Objects come from the bundled JSON pack; only pixels ever needed g1.dat. That makes the benchmark legally shippable in CI, which was the point: this is the prerequisite for running CoasterBench inside vLLM's CI as an end-to-end agentic tool-calling eval.

  • Game: --no-graphics flag; capture/render paths refuse cleanly; the MCP server masks its image tools server-side regardless of client-advertised modalities. Assetless scenario default: the checked-in BigMapTest.sv6 (terrain probed: open flat grass, cash-rich).
  • Graphics without assets: the program executor records a per-piece cursor trace into report.json; the driver renders it as a two-panel schematic PNG (top-down + isometric, station/chain-lift/height color-coded, dashed gap-to-start line for open circuits). Every round gets a track.png; --schematic-feedback feeds it to multimodal contenders.
  • Driver: --base-url speaks to any OpenAI-compatible endpoint (vLLM, llama.cpp, …) through an Anthropic-shaped facade, plus everything real serving forced: retry on contract violations, --max-tokens with fail-fast forensics (failed calls dump their raw response), --chat-template-kwargs, preserved reasoning passback across turns, in-band submit forcing, hour-long timeouts, COASTERBENCH_CLI/COASTERBENCH_OPENRCT2_DATA env overrides for containers. Default 6 rounds.
  • CI packaging (evals/ci/): multi-stage Dockerfile — lean game image (--target runtime, ~240 MB) and a self-contained eval-runner image (game + driver + preinstalled deps); check_run.py protocol-success gate; README with the vLLM job shape and field notes.

Verified

  • macOS, Linux/arm64 (podman), and Linux/x86_64 (in-cluster buildah): identical ratings across all three.
  • Hermetic x86_64 smoke (no network: caught and fixed the objects-at-wrong-prefix install bug that runtime object auto-download had been masking).
  • Full in-cluster end-to-end: the containerized driver played a real model (Laguna-S-2.1 on 8×A100 vLLM) pod-to-pod, 2 rounds, standings written — the complete CI topology in 3m40s including image build.
  • Live runs against vLLM 0.25.1 (Qwen2.5-7B, Laguna-S-2.1): field notes in evals/ci/README.md document one confirmed serving bug (named tool_choice unenforced on poolside_v1, 3/3 repro), one unconfirmed anomaly, and the reasoning-model verdict (thinking never terminates on the one-shot prompt at any budget up to 131k — thinking-tier models need the interactive MCP mode).

Trade-offs

Assetless runs are text-only-plus-schematics (no sprite screenshots) and have no stock TD6 library, so the similarity penalty is inert — report.json says similarity: null and run.json records no_graphics: true; such runs are not leaderboard-comparable with asset-full ones.

Follow-ups (in flight separately)

Scenario fleet (seeded --make-park generator, ~100 parks, concurrent eval), goal variety (guest-services/stalls, finish-the-coaster), and the companion vLLM-side branch (tests/evals/coasterbench/ + optional Buildkite step).

🤖 Generated with Claude Code

tlrmchlsmth and others added 18 commits July 24, 2026 17:07
@tlrmchlsmth @claude
No-assets mode: run the whole eval without RCT2 files (--no-graphics)
ac3978b 
Everything the eval scores — park load, placement, testing, ratings, the
drawability gate — is pure game logic; only pixels need g1.dat. Proven
in-tree by upstream's own RideRatings tests, which load parks and assert
exact ratings with NoGraphics set in an assetless CI.

- eval --no-graphics: skips LoadBaseGraphics, refuses --capture/
  --render-library combos; default (graphics) behavior unchanged.
- orct2_host_capture guards cleanly; new orct2_host_graphics_available
  crosses the ABI (header regenerated).
- MCP server intersects IMAGE out of every request's modalities when
  sprite data is absent: screenshot/best_screenshot vanish from
  tools/list and refuse calls regardless of what the client advertises;
  finish_and_test skips the best-shot capture instead of logging errors.
- Assetless scenario default: the checked-in BigMapTest.sv6 (mostly open
  flat grass — probed — cash-rich, builds and tests fine). Verified
  end-to-end with zero RCT2 assets on this machine: bpb.sv6 rates 134
  rides; a 25k-tick eval takes ~7s; a real 148-piece run program behaves
  identically to the current asset-full build.
- driver.py: --no-graphics lane (design mode only, report-only feedback)
  and --base-url for any OpenAI-compatible endpoint (vLLM serve etc.) via
  an Anthropic-shaped facade over chat completions; exercised against a
  wire-format-asserting stub for two full rounds including validate.
- evals/ci/: CPU-only Dockerfile (draft, unexercised on Linux),
  check_run.py protocol-success gate (validated against existing runs),
  and the nightly-job shape for running under vLLM CI.

Similarity penalty is inert without the stock library (report records
similarity: null, run.json records no_graphics) so assetless scores are
not comparable with asset-full leaderboards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI image: build the orphaned graphics target; verified on Linux arm64
c0c1667 
DISABLE_GUI leaves the graphics target (g2/fonts/palettes/tracks.dat,
generated by the sprite compiler from in-repo resources) with no
dependents, but the install manifest still expects the files — build it
explicitly before install.

First Linux build of the fork: 237 MB image, containerized MCP server
places/tests/rates a coaster with ratings identical to the macOS build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
OpenAI lane: retry when tool_choice=required returns no tool call
c5666e8 
Observed once against vLLM 0.25.1 (Qwen2.5-7B, hermes parser): the
server's first structured-output request after startup answered with
zero tool calls, violating the required contract. Not reproduced in 15
probes or a full 4-round run, so treat it as rare and retry (3 attempts)
instead of killing the run; usage from wasted attempts still counts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: --max-tokens, fail fast when thinking exhausts the budget
6da1d26 
Laguna-S-2.1 (reasoning model, poolside_v1 parsers) burned all 8000
completion tokens thinking and never reached a tool call — finish_reason
"length", surfaced as a tool_choice=required violation. Retrying that is
deterministic waste, so the OpenAI lane now raises immediately with the
real cause, and the per-request budget is a flag (default unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI notes: document the vLLM findings from the first live runs
cb35fe7 
Three failure classes surfaced on day one against vLLM 0.25.1 (all
handled by the driver, first two worth an upstream look): a cold-start
tool_choice=required contract violation, reasoning models exhausting
max_tokens before the tool call, and generation_config.json silently
overriding sampling defaults.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: named tool_choice is advisory on some stacks; retry forced su…
b122f5e 
…bmit

vLLM + poolside_v1 returned validate_track_program when the request
forced submit_track_program by name (finding #4 for the CI notes), so
the final forced step of a round now gets three attempts, answering
whatever tool the model actually called in between. Also thread
chat_template_kwargs through the OpenAI lane (--chat-template-kwargs)
so reasoning models can run with thinking disabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: COASTERBENCH_CLI env override for CI binary location
b952e70 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI notes: final attribution — one confirmed vLLM bug, one anomaly, on…
6d66daa 
…e harness fix

Named tool_choice unenforced on poolside_v1 is now 3/3 reproducible; the
required-zero-calls event stays unconfirmed (no captured body); the
max_tokens starvation was ours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: force submission in-band when the validation budget runs out
81da8a0 
With named tool_choice advisory (finding #1), agentic models keep
validating through all forced steps and never submit. The last budgeted
tool result now says so explicitly, which instruction-following models
respect even when grammar enforcement doesn't exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
Asset-free track graphics: cursor trace in the report, schematic PNG …
5911936 
…per round

program.rs records the cursor after every placed piece into
report.json's program.trace; the driver renders it as a two-panel
(top-down + isometric) diagram — station green, chain lift red, height
gradient, and an explicit dashed gap-to-start line for open circuits,
which was today's dominant failure mode and invisible in prose.
--schematic-feedback attaches it to round feedback for multimodal
contenders; the PNG is always saved as a round artifact regardless.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI image: bundle the smoke program alongside the scenario
6dbbff5 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
OpenAI lane: preserve reasoning across turns
421cf25 
vLLM renders `reasoning` passed back on assistant messages into the
chat template (verified via prompt_tokens delta), so thinking models
keep their derivations instead of re-reasoning from scratch each call —
the difference between bounded per-step thinking and 60k-token restarts.
Captured from the SDK's extra fields, replayed by _to_openai; inert for
non-reasoning models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: default to 6 rounds
f50817c 
Best-across-rounds is the score; models rarely close a circuit early and
more rounds help roughly monotonically. CI overrides to 2 for time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI image: fix objects prefix bug; add self-contained eval stage
6016489 
The objects/asset downloads run from install(CODE) strings that bake
CMAKE_INSTALL_FULL_DATADIR at configure time, so installing with
--prefix /usr left objects at /usr/local — present in the image,
invisible to the binary. (Runtime object auto-download masked this on
any container with DNS, which is how the arm64 e2e passed.) Prefix now
set at configure. New default `eval` target layers python venv + driver
on the game so one container runs a whole benchmark; game-only image is
--target runtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: COASTERBENCH_OPENRCT2_DATA env for containerized data path
4e7630d 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
OpenAI lane: hour-long request timeout for thinking models
4abde85 
The SDK's 10-minute default killed a legitimate 85k-token think
mid-generation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
CI notes: thinking-tier verdict — unbounded reasoning on one-shot pro…
c1c8d47 
…mpts

Every confound removed (512k context, 131k budget, hour timeout,
pod-to-pod network, reasoning passback); turn-one thinking still never
terminates, with or without tools. One-shot runs disable thinking;
thinking models belong in the interactive per-piece mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
@tlrmchlsmth @claude
driver: save the raw response when a call exhausts its budget
ea30482 
The 131k-token thinking trace was the evidence both times a run died
this way, and raising without persisting it lost it both times. Failed
calls now dump the full response into the run dir.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tlrmchlsmth