Gate production behind a deliberate release, and check a real model actually produces geometry

[kmatzen]

The two things left open from the goal. Both are process/verification rather than features.

1. Merging to main was a production release

There was no gate between "PR approved" and "live".

Changing it surfaced a second problem I hadn't spotted: deploy.yml and ci.yml both triggered on the same push and ran concurrently. Nothing made the deploy wait for the tests, and this build is faster than the e2e suite — so in practice production was published while main's own CI was still running. A red main could reach users, and usually would have.

Publishing is now:

git tag v1.4.0 && git push origin v1.4.0     # leaves a record of what shipped
gh workflow run deploy.yml --ref main         # or publish by hand

Neither implies tests ran — a tag can point at any commit — so a guard job looks up the completed CI run for that exact SHA and refuses anything that isn't success, including "no CI run found". An untested commit can't be published by tagging it.

Verified the guard's query against the live API before trusting it:

SHA	result
560b514 (green main)	success → deploys
0000000 (nonexistent)	empty → fails closed

Rollback is the same mechanism against an older ref; the guard allows it because any previously-shipped commit has its own green CI. DEPLOY.md updated, including why the push trigger shouldn't be restored.

This changes your workflow — merging no longer ships. That's the point, but it's the one thing here you may want to feel out before merging.

2. Nothing verified a real model produces geometry

The wire is tested, the parser is tested, and chatStore.send.test.ts joins them with both stubbed. So the entire suite can be green while chat produces nothing printable — which is how two bugs got to a user this year rather than to the suite.

src/llm/live.test.ts uses the real system prompt, real wire and real parser, and asserts on evaluated geometry rather than JSON shape. A reply can parse perfectly and still describe an empty union or a shape at the wrong scale.

• "Make a 20mm cube" → bounds 20 mm on every axis. The falsifiable one; also catches the prompt failing to establish millimetres, which is how this goes subtly wrong rather than obviously wrong.
• An L-bracket → samples inside its own bounds are inside the solid, on a 5×5×5 grid rather than the centre, because an L's centre is fresh air.
• An edit to an existing box → reaches the branch of the system prompt that serialises the current tree, which no first-turn test touches, and checks the untouched axis stayed put.

Verified it isn't vacuous — a skipIf test that never runs is worse than no test. With a deliberately bogus key, all three fail with a genuine 401 from OpenRouter, so they really do reach the network:

LLMAuthError: OpenRouter rejected the connection (401)  {"error":{"message":"User not found.","code":401}}
 ❯ streamOpenAI src/llm/llmService.ts:297:21
 ❯ ask src/llm/live.test.ts:53:16

Default npm test: 503 passed | 3 skipped, green.

Runs weekly, not per-PR — what breaks this path is a provider changing its wire format or a model drifting out of the requested format, neither of which our commits trigger, and a paid third-party-dependent check on every PR gets disabled within a week. The job fails loudly if the secret is missing, so an absent credential can't read as a pass.

What you need to do

To turn on the weekly run, add a repo secret LIVE_LLM_API_KEY (OpenRouter by default). Until then the scheduled job will fail deliberately rather than pretend. You can also just run it locally:

SINTER_LIVE_API_KEY=sk-or-... npm run test:live

I can't run it against a real model myself — that needs your credits — so this PR proves the harness works, not that the feature does. The first real run is yours.

🤖 Generated with Claude Code

[github-actions]

🚀 Preview deployed: https://ops-deploy-gate-and-live-llm.sinter.pages.dev

(updates on every push to this PR)