fix(experiment): judge preflight + failed-run data preservation - #68
Merged
Merged
Conversation
…002 attempt-1 post-mortem) Run-002 attempt 1 failed: OPENWEIGHT_API_KEY present but INVALID (Together 401 on every call). Two design gaps let that sink the run: 1. --skip-missing only checked env-var PRESENCE, so the broken judge burned 1252 failed calls, pushed the failure rate to 33% > 20%, and the CLI (correctly) refused to bless the run. NEW: preflightJudges() makes ONE live probe call per judge before its ~1250 scoring calls; any preflight failure (invalid key, wrong model id, unreachable endpoint) skips that judge and records it as a protocol deviation — the run proceeds with the judges that actually work. All-fail → loud refusal with a checklist message. 3 new tests. 2. The workflow's publish step did not run after a failed experiment step, so the ~2500 PAID Anthropic+OpenAI calls of attempt 1 were written to the runner disk and lost. NEW: publish runs if: always(); a failed run's data is pushed on an experiment/<run>-FAILED-<id> branch for diagnosis — with NO automatic pull request (a failed run must never be offered as results), and a no-output guard when the run died before writing anything. Diagnosis note recorded: OPENWEIGHT_BASE_URL was CORRECT (the 401 is Together's own response body — the endpoint was reached); the API key secret itself is invalid and must be replaced. Validation: web 414/414, core 213/213, cold tsc clean, sync no drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Post-mortem of run-002 attempt 1 — and the two fixes so it can never happen again
Attempt 1 failed with
OPENWEIGHT_API_KEYpresent but invalid (Together returned 401invalid_api_keyon every call). Diagnosis from the logs: the base URL was correct — the 401 is Together's own response body, so the endpoint was reached; only the key is bad. Two design gaps turned one bad secret into a lost run:1. Judge preflight (
preflightJudges)--skip-missingonly checked that env vars exist. The broken judge then burned 1 252 failed calls, pushed the failure rate to 33 % > 20 %, and the CLI (correctly, per its honesty threshold) refused to bless the run. Now: one live probe call per judge before its ~1 250 scoring calls. Any preflight failure — invalid key, wrong model id, unreachable endpoint — skips that judge and records it as a protocol deviation (same mechanism as Run 001's missing-key skip); the run proceeds with the judges that work. All judges failing → loud refusal with a checklist message. Cost: one ~8-token call per judge. 3 new tests.2. Failed-run data preservation
The workflow's publish step didn't run after a failed experiment step, so attempt 1's ~2 500 paid Anthropic+OpenAI calls were written to the runner disk and lost. Now the publish step runs
if: always(): a failed run's data is pushed to anexperiment/<run>-FAILED-<id>branch for diagnosis — with no automatic PR (a failed run must never be offered as results) and a guard for runs that die before writing anything.Validation
web 414/414, core 213/213, cold tsc clean, sync-core no drift. Workflow changes are dispatch-only paths; no CI-path changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
Generated by Claude Code