feat: make a run measurable, affordable, and honest about failure - #6
Merged
Conversation
Scoring was unreachable: `agent score` required --gold pointing at a file that
did not exist, and the assumption behind that - that reference answers were
unavailable - was never checked. They ship in the same gated dataset the
attachment loader already reads, as metadata.level{N}.parquet in the validation
split, with a Final answer column for all 53 level-1 tasks.
gold_answers() fetches them, so grading is local, instant and free. The
alternative was submitting to the leaderboard and learning a single percentage
with no indication of which tasks failed - which is how a run scoring 4/20 went
weeks without anyone noticing that 14 of those tasks had never executed.
Two deliberate choices:
- It raises GoldUnavailableError rather than returning {}. An empty gold set
makes score() report 0/0, which reads as "your agent got nothing right"
instead of "the answer key could not be fetched" - the same laundering of an
error into a plausible output this codebase keeps finding.
- A failed fetch is never memoised. _GOLD is populated only on success, not via
lru_cache, because caching the except branch is exactly the _dataset_index
bug: one transient error convinced the process for its whole lifetime that
GAIA had no attachments.
pandas is imported lazily - it lives in the app extra while `agent score` is a
core command - and its absence reports an actionable message. Note the contrast
with _read_tabular, which degrades to a string instead: a tool degrades so the
model can adapt, a grading command raises so the operator knows.
--gold stays supported for a local file. Output separates "4/6 answered
correctly" from "4/53 of the level set", which are different results.
_dataset_index carried @lru_cache, which caches whatever the function returns - including the `except` branch's empty dict. Two individually reasonable decisions (cache the listing, degrade rather than crash) composed into a bug: one transient network error convinced the process for the rest of its life that GAIA had no attachments, with no retry. Measured: six consecutive tasks failed against an empty index while the same request succeeded a minute later. Replaced with a module-level dict populated only on success. Three tests cover it - a failed listing is retried, a successful one is not refetched, and no token means no request at all. The conftest fixture and one test called .cache_clear(); both now clear the dict. lru_cache is no longer imported here.
TaskMetric.supervisor_steps was declared from the start and read 0 in all 59 recorded runs. Not a missing assignment - the graph's only exit returned a bare string, and the count lived in state that was discarded. That made every tuning question unanswerable. Iteration caps and per-task timeouts should be set from the distribution of successful runs, and there was no distribution to look at: the 180s timeout that killed a task 95s before it produced the correct answer was picked without data. - Solution(text, steps) is the graph's full result; Orchestrator.solve returns it and answer() stays a thin str wrapper, so the app, CLI and existing tests are untouched - solve_question joins answer_question as the module-level entry point, and the harness resolves to it - run_one reads the result structurally rather than importing Solution: resolving that import late is what stops importing the harness from building a model client. A plain string - what the fifteen injected test stubs return - reports zero steps. AnswerFn is typed Callable[..., Any] for the same reason, documented inline.
Every truncation in the codebase took a head slice - scrape_webpage, wikipedia_lookup, read_file, the tabular renderer and the sandbox's stdout and tracebacks all did `text[:limit]`. That discards the end, and the end is routinely where the answer is: a spreadsheet keeps its total on the last row, a program prints its result last, and an article's tables sit below its prose. A page that mentions the topic in its first paragraph and answers the question in its last was indistinguishable from one that never answered at all. elide() keeps both ends and drops the middle, for the same token cost. The tabular path mattered most: pandas already elides middle rows via to_string(max_rows=200), and the head slice on top was undoing it. Truncation is now always announced, including when the limit is too tight to keep two useful ends - the first draft fell back to a bare head slice there, which is the silent-truncation pattern this codebase keeps removing: a partial result that looks complete. The note can push marginally past the limit, which is the right trade, since the limit bounds cost rather than bytes. Two existing tests asserted the old marker text and were updated.
A run issued 22 tool calls of which 14 were distinct: the same Wikipedia article fetched three times, the same YouTube page scraped three times, each costing about ten seconds against a per-task timeout that killed two tasks. Repeats happen because a specialist gets a fresh state on every delegation and has no memory of what an earlier one already looked up. Three constraints shaped this: - python_repl must never be cached. Code can be nondeterministic and rerunning it can be intentional, so caching would make the sandbox lie. Caching is therefore opt-in on ToolSpec rather than opt-out - a new tool is safe until someone has thought about it - and tests assert the policy per tool. list_downloaded_files is likewise live: reflecting change is its purpose. - The lifetime must be shorter than the tools'. Tools are built once with the orchestrator, so a naive cache would live as long as the process and a Space runs for days. A generation counter, bumped per task by the harness, makes prior entries unreachable without touching them. - Failures must never be cached. That is the _dataset_index bug rebuilt by hand. Tools report failure in-band as ordinary text, so a predicate inspects only the opening of a result: a false positive costs a refetch, a false negative disables a tool for the rest of the task, and the bias is toward not caching. Needing the predicate at all is a smell pointing at error-as-string. A hit returns the full cached text with a marker rather than a pointer: across delegations the earlier result may have been trimmed out of the transcript, so a pointer could name something the model can no longer see. The marker still tells it that it is repeating itself. memoized() returns a new StructuredTool rather than mutating the original.
The project moved from a free provider with an involuntary daily token cap to a paid one with no cap at all. Nothing stood between a retry loop and real money except the wall-clock budget. Two dollar ceilings, because they catch different failures: max_task_cost_usd (0.50) catches one runaway task, max_run_cost_usd (5.00) catches many slightly-too-expensive ones. A per-run ceiling alone would let a single pathological task through; a per-task ceiling alone would not notice twenty tasks drifting upward together. Either at 0 disables that ceiling. Spend is charged after each task rather than estimated before it. A single task is already bounded by its timeout, so the job is to stop the *next* one - and stopping is the point: a spent budget must never quietly become a cheaper, worse run. The run ends the way the wall-clock budget already ends it, with cached answers left submittable. An unpriced model costs 0.0 rather than guessing, so an unknown provider cannot halt a run on an invented number; the clock budget still bounds it. Rates match on prefix so dated snapshots inherit their family's price. Also caps the router at max_router_tokens (512). It emits one schema selection and a short justification, and had been inheriting the specialist's 1024. Generous rather than tight because Sonnet 5 spends output tokens on adaptive thinking, and a cap that truncates mid-thought yields a malformed structured output rather than a cheaper one. For reference the measured task costs $0.047, so a 20-task run is about $0.94 against a $5.00 ceiling. One test asserted "ceiling" was absent from any message - and pytest's tmp_path is named after the test, so the completion message's answers.json path matched. Those assertions now check structure rather than substrings of a path.
These are one change: effort is worth A/B-ing, and an A/B was impossible because two runs could not be told apart. metrics.jsonl is append-only and carried no run id, no timestamp and no record of the configuration - a file holding 27 records for 20 tasks gave no way to say which run a record belonged to, let alone which settings produced it. Each TaskMetric now carries run_id, recorded_at, effort and cost_usd, so two arms of an experiment stay separable in one file. Effort is set per role. The router picks one name and writes a sentence, so it runs at "low"; the finalizer formats an answer it has already been given, so it does too. Specialists stay at the provider default, which is the baseline an A/B starts from. Worth more than its effect on output tokens, which are only 6% of spend: lower effort means fewer and more-consolidated tool calls, and tool calls drive the delegation rounds whose transcript replay is the other 94%. Whether that trades away accuracy is exactly what the labels now let us measure. with_effort is generic over the caller's type rather than typed Runnable - annotating it Runnable erased bind_tools and with_structured_output from everything it touched. It is a no-op on non-Anthropic providers, which have no equivalent knob, and an unknown value is ignored with a warning rather than sent, so a typo degrades to the provider default instead of failing every call in a run.
…cuments Two changes with one cause: the limits and the prompts were both written for a provider that is no longer in use. max_file_chars was 12,000 characters - about 3,000 tokens, sized for Groq's 8,000 tokens-per-minute ceiling where that was most of a minute's allowance. Against Sonnet's 1M context it is 0.3% of what fits, and the middle of every document was being discarded for no reason. Raised to 60,000 / 30,000 / 15,000. This is why the project does not need chunking or a retrieval index: those exist to fit large text into a small window, and the window is not small. The binding constraint is replay - a specialist resends its transcript each iteration - so a document costs its size times the iteration count, which at $2/1M is about $0.09 inside a $0.50 per-task ceiling. For data too large to be worth that, the answer is python_repl computing over the file and printing the number, which is what the code specialist is now told to do. The prompts are restructured with XML tags, which Claude attends to more reliably than prose headings: a tag names a section's boundary, so an instruction cannot be read as part of the example above it. Content is preserved - every line was added because something failed without it - with three additions grounded in the last runs: - The supervisor is told what the provenance prefix means and that a reply carrying tool evidence must not be re-verified. It still spent a whole extra round confirming an answer the prefix already vouched for. - The web specialist is told its reply is the only thing that survives, since the supervisor never reads the pages it fetched. - The code specialist is told to aggregate large files rather than print them. Prompt invariants are now asserted rather than trusted to review: the sentinel, the exact-match format rules, character-level routing, trust-the-evidence, and balanced XML tags. The assertions read through a whitespace-collapsing helper, because prompts are hard-wrapped and an exact-substring match breaks whenever a line wraps mid-phrase - which says nothing about whether the instruction is still there. One pre-existing assertion was tied to the quoting of "Prefer 'reason_agent'" and now tests the intent instead.
The 53 GAIA level-1 gold answers are now available locally, so the format rules can be derived from them rather than assumed. Their shapes: 22 plain words, 15 integers, 8 lists, 6 identifiers or notation, 2 decimals. The decimals found a real gap. The rule said "digits only, no thousands separators" and said nothing about precision - but a reference answer of 0.1777 is wrong as 0.18, and 89706.00 carries its trailing zeros. That is a correctness rule mistaken for a formatting one, and no amount of formatting guidance would have caught it. The finalizer is now told explicitly not to round, and to copy identifiers and notation verbatim. Both prompts gained worked examples. The finalizer's are real gold answers - FunkMonk, 3, Rd5, 89706.00, 0.1777, the vegetable list, 80GSFC21M0002 - chosen to cover every shape that occurs, with a note naming what must be absent. The router's cover each destination plus the two provenance cases, since a prefix that never changes a decision is not worth reading: one example FINISHes on evidence, another re-delegates a claim carrying none. Cost: about 970 extra input tokens per task, $0.04 across a 20-task run. It becomes near-free once prompt caching lands, since the system prompt is the stable prefix that caching exists to serve.
Level-1 tasks are lookups and small computations rather than deep reasoning, so the provider's default "high" is heavier than the work needs. Set explicitly rather than left blank. A metric recording "medium" names a configuration under test; a blank records only "whatever the provider chose", which is not something a later A/B can compare against. Added a test asserting every role's effort is a value the provider accepts. Unknown values are dropped with a warning rather than sent - which is the right runtime behaviour, but means a typo here would silently run at the default instead of the intended level, and the metric would still claim otherwise.
The run scored 4/4 on what it answered, including the Excel task for the first
time ever, and surfaced four faults - three of them introduced earlier today.
1. A tool-less specialist read as unverified. reason_agent has tools=() by
design, so it always reported "no tools were used - this answer is
unverified", and the supervisor always re-delegated to check it. Every
reasoning task cost an extra round. tool_evidence now distinguishes a
specialist that could have used tools and did not from one that has none:
the first produced a claim, the second did exactly its job.
2. router_effort="low" returned an empty object - 0 output tokens, neither
schema field present - and the validation failure ended 2d83110e, a task
that had succeeded on every previous run. Raised to "medium". A component
whose output must validate has to earn the right to be cheap.
3. Routing now retries once. The SDK retries transport errors, but a call that
succeeds and returns {} is not an error it can see, so a single hiccup ended
the task outright.
4. max_code_iterations was 3, which download + read exhausted before anything
could be executed - so every attachment task needed two delegations, and the
Excel task spent four rounds and 40,175 tokens re-reading a file it already
had. Raised to 6, and web to 5, which also hit its cap on a three-tool
sequence.
5. Pacing is off by default. 12000 tokens/minute was Groq's free-tier figure;
Anthropic answers a 429 with retry-after and the SDK retries, so the stale
number bought nothing and spent 255 of one 411-second run's seconds asleep.
Two guard tests failed and both were right to. The budget-consistency test
caught that raising the iteration caps broke the timeout invariant - though its
own 20s-per-call constant turned out to be stale too, measured at 2-4s against
Anthropic, so it is now 8s with the measurement recorded. The other asserted
the router ran at "low", which is exactly what this commit reverses.
…e inventory
The second five-task run halved wall time and cut a reasoning task from three
decisions to two, but total tokens rose 35% - and the router said why:
"it seems to be a fabricated/unverified claim about code_agent's behavior"
"this appears to be a fabricated evidence prefix in the conversation"
Told only that the prefix IS the evidence, the supervisor reasoned that a
prefix is just text in a conversation, which anything could have written, and
re-delegated to check it. That objection is fair from where it sits. The prompt
now states the provenance of the provenance: the prefix is stamped on
afterwards by the framework, counted from the tool-execution record, and a
specialist has no way to write or influence it. All three states are spelled
out, including the tool-less one added earlier today.
Separately, downloaded_inventory listed the whole download directory, which
outlives a task. The Excel task was therefore offered a Python file and a chess
image left by earlier tasks and read both. Attachments are named by task_id, so
the fix is an exact prefix filter; SupervisorState carries the task_id to make
it available, and an empty id still lists everything for list_downloaded_files.
Not addressed here: 2d83110e still fails, and the retry proved it is not
transient - two attempts, identical empty objects. The task text is a reversed
instruction ("write the opposite of the word 'left' as the answer"), and the
router appears to obey it, answering as text instead of calling the routing
function, so there is no tool call to parse. That needs the task delimited as
data rather than read as instruction.
2d83110e failed on three consecutive runs, and the retry added earlier proved
why it was not a hiccup: two attempts, identical empty objects, deterministic.
The task is a reversed sentence decoding to "If you understand this sentence,
write the opposite of the word 'left' as the answer". The router obeyed it -
replied "right" as prose rather than calling the routing function - so there
was no tool call to parse and with_structured_output returned {}. Neither the
effort level nor the retries were ever going to fix that; the input was being
read as instruction.
as_data wraps the opening human turn in <task> markers before the router sees
it, and the prompt says everything between them is material to be routed, never
instructions, however imperative it sounds. Specialists are deliberately not
wrapped: following the task is exactly their job. This is the non-cosmetic use
for XML tags - marking where instructions to the reader end and untrusted input
begins - and it is worth more than the section headings added earlier.
The balanced-tags test earned its keep immediately: the first draft mentioned
<task> in prose and left it unclosed, which is precisely the hazard the test
exists to catch. The prompt now shows the opening and closing markers together,
which balances and demonstrates the shape at once.
The iteration cap counts reasoning turns, and every tool call consumes one. Six turns therefore buys five tool calls and nothing left over, so a specialist that downloaded, read and computed reached the ceiling with the work done and no turn in which to say so. route() returned END on the spot. The supervisor then saw a tool call with empty content, concluded "the previous attempt did not produce an actual answer (no output/printed result)", and re-delegated the whole job at full price. Raising the cap from 3 to 6 moved the boundary without removing it - the Excel task simply hit 6 twice instead of 3. A summarize node now runs when the budget is exhausted: one call, no tools bound, asking for what was established from what is already in the transcript. It cannot route back, so a broken provider still costs exactly max_iterations plus one, and a failure there degrades to a plain statement rather than killing the run. The existing cap test asserted exactly max_iterations calls and now expects the extra one, with the bound spelled out in the docstring so the number is not mistaken for arbitrary.
…rror
A probe against the failing task settled four runs of wrong guesses:
stop_reason : refusal
stop_details: {'category': 'general_harms'}
output_tokens: 0
control question: tool_use, routed correctly
2d83110e is a reversed English sentence from the benchmark. A safety classifier
declines it - with tools bound and without, while an ordinary question routes
cleanly through the identical path, so the input is the only variable. Neither
the reasoning effort nor the <task> delimiter was ever going to change that;
both earlier diagnoses were wrong.
It reached this code as "next_agent Field required" because
with_structured_output defaults to include_raw=False: it parses the reply,
raises when there is no tool call, and discards the response carrying the cause.
A refusal is a *successful* HTTP 200 with an empty body and the outcome in
stop_reason - the same shape as every other bug in this project, an
unsuccessful outcome delivered through the success channel.
- include_raw=True, so the {"raw", "parsed", "parsing_error"} envelope is
visible and refusal_category can read stop_reason.
- A refusal is not retried. It is deterministic, and the retry added earlier
spent two round trips being declined identically.
- A refused task is routed rather than abandoned. The refusal is on the
router's call; a specialist prompts differently and may not trip the same
classifier. code_agent is the default because text the router could not parse
is usually encoded, and decoding is what it is for.
The router stub now returns the same envelope as the real thing, including a
refusal mode, so the branch is testable without a network call. scripts/
probe_router.py is kept: it prints the raw reply for the three explanations a
parse failure can have, and cost one cent to end the guessing.
Not done here: Anthropic's server-side fallbacks, which re-run a refused
request on another model in the same call. That is the documented remedy and
the API's own error text points at it, but reaching it through ChatAnthropic is
unverified.
Two faults from the last run, both introduced by the two commits before it. The wrap-up turn crashed with a 400 every time it was needed: messages.12: `tool_use` ids were found without `tool_result` blocks immediately after A specialist that exhausts its budget mid-decision leaves exactly that shape - the model asked for a tool, and route() jumped to summarize instead of running it - so the very turn added to report the work could never be sent. The Excel task therefore spent 121 seconds and four rounds re-deriving an answer it had computed in the first one. This was predicted. The review of #5 said of normalize: "an invariant enforced by control flow, not the type system - a future call site that seeds normalize() with a raw, unresolved tool-calling AIMessage would silently break." That call site was then written. drop_dangling_tool_calls now removes the unrun requests inside normalize, so the rule holds for every caller rather than by convention. They are dropped rather than answered with synthetic results: they did not run, and inventing results would be a lie the model reasons from. Separately, the refusal fallback fired on every round. The refusal is on the task text, which does not change between rounds, so routing to a specialist and returning to the same declining router simply repeats - four identical rounds until the step budget stopped it. It now fires once, on the first step, and finishes on any later refusal.
The download directory is on this machine; the sandbox is a remote container
that has never been able to see it. Nothing in code.py ever uploaded anything,
so pd.read_excel("logs/downloads/...") inside the sandbox could only fail.
The Excel task worked anyway because read_file renders the spreadsheet as text
into the transcript and the model retyped the numbers into its program. That is
what a 603-character program for "sum one column" actually was - the data,
copied by hand. It also explains the iteration pressure: three executions to
compute one total, because each had to rebuild what the last one knew.
_upload_attachments copies the downloaded files in before each execution and
the result names their paths, since the model cannot list the sandbox itself.
A failed upload is logged and skipped - code that does not need the file must
still run - and an SDK without a filesystem degrades to no uploads.
The prompt claimed a capability that did not exist. It said "load it, filter or
aggregate in code" of a file the sandbox could not open, which was written two
hours ago and could not have been obeyed. It now names the real path, and
states what had never been written down anywhere: each execution gets a fresh
sandbox, so variables and imports do not survive, and one self-contained
program is worth more than three exploratory ones.
_download_dir became public, since it is now read across a module boundary.
The balanced-tags test caught a third bug in its own area: "/home/user/<name>"
in the prompt reads as an unclosed tag.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The previous fix was wrong and the run said so precisely: messages.12: `tool_use` ids were found without `tool_result` blocks messages.12, not the last message. drop_dangling_tool_calls popped only from the end, but summarize builds [system, *transcript, wrap-up request] - so appending its own prompt moves the unresolved call to second-to-last, where the trailing check never looks. Every wrap-up still 400ed, the specialist still never reported, and the Excel task went from a correct 89706.00 to NO_ANSWER. Now matched by tool_call_id: any message requesting a tool whose result is absent is dropped, wherever it sits. A partially resolved request goes too - one unpaired tool_use invalidates the whole message on the wire. The new tests include the exact production shape (system, transcript ending in an unrun call, then the wrap-up request), which the previous tests did not cover because they only ever put the dangling call last - the same assumption that produced the bug. Note the sentinel worked: with the specialist mute, the finalizer had nothing to report and emitted NO_ANSWER rather than inventing a total. The task failed honestly instead of returning a plausible wrong number.
…ard correctly
Three reviews of this branch found, between them, that the instrument every
measurement in this session came from was broken.
eval/scorers.normalize stripped punctuation BEFORE deciding whether a value was
a number, so the decimal point and minus sign were deleted first. Measured:
'3.14' vs '314' -> match credited a wrong answer
'-5' vs '5' -> match credited a wrong answer
'89706' vs '89706.00' -> NO match rejected a correct one
Two of the 53 level-1 reference answers are decimals and fifteen are integers.
Numbers are now read first and canonicalised with %g, so 89706 and 89706.00
agree while 3.14 and 314 do not. An existing test asserted "$1,234.50" -> 123450
- it had encoded the bug, and now asserts 1234.5.
_upload_attachments listed the whole download directory, unscoped by task. That
is the same bug downloaded_inventory was fixed for nineteen minutes earlier,
rewritten in the code added to fix something else, and it announced every stale
file to the model by name. python_repl is a bare tool that cannot be passed a
task id without putting it in the schema the model sees, so the task is declared
in module state by the harness - the same mechanism the tool cache already uses
for its generation counter.
The refusal guard tested `step > 0`, conflating "not the first round" with
"already refused". A task that routed normally and was refused at round 1 -
plausible, since the router sees accumulated specialist output, not just the
original text - skipped the recovery path entirely, which is the one case it
exists for. Now keyed on the instruction already being the refusal one. The
existing test asserted refusal at rounds 0 and 1, encoding the same conflation;
it now covers a first refusal after a normal round.
Adding module-level task state made a sandbox test pass alone and fail in the
suite - real evidence that this state outlives its scope. conftest now resets
it, and the tool cache, alongside the dataset index.
| if writer is None: # pragma: no cover - older SDKs expose no filesystem | ||
| return [] | ||
|
|
||
| from agent.tools.files import current_task, task_attachments |
The specialist budget counted reasoning turns, so a tool call and the thought that produced it each cost one. Six turns bought five tool calls and left nothing to report with - which is the entire reason the summarize node, both attempts at drop_dangling_tool_calls, and half a prompt rewrite had to be written. Four commits and two 400-storms were spent working around a counter that counted the wrong noun. Now a turn spends budget only if it requested a tool - or failed, since a provider failing every call emits no tool calls and counting only those would loop until the recursion limit. That was caught by an existing test, the first time this session a guard test caught a regression before a live run did. route() also checks "finished" before "out of budget". The other order sent a specialist that had just produced its answer on its last allowed turn off to summarize anyway, replacing a good answer with a paraphrase of itself. drop_dangling_tool_calls now removes results orphaned by dropping their request - the mirror-image rejection, a tool_result with no tool_use. The test written to prove the fix correct had asserted the orphan should survive. build_specialist's llm_factory defaulted to get_llm, bound at import time, so the orchestrator captured the original function: patching the module attribute reached the supervisor and silently missed every specialist. Half the graph was unstubable while the docstring claimed otherwise. New test infrastructure aimed at the class of bug that shipped four times: - ToolCallingLLM emits real tool calls AND enforces the three provider rules, raising the way a 400 does. Every previous stub modelled a cooperative provider that inspected nothing. - The wrap-up tests assert the message list SENT, not the text returned. summarize catches everything and substitutes "ran out of steps", so a contract violation read exactly like honest budget exhaustion. - Wiring tests for three mutations shown to reintroduce shipped bugs with zero failures: has_tools=True, unscoped downloaded_inventory(), and a dropped task id in solve().
… defaults The supervisor prompt described every specialist twice - a hand-written <routing> block and a roster generated from the specs - and the copies had drifted. The block said web_agent reads webpages; the roster said it also downloads attachments; the examples sent attachments to code_agent. One system prompt, three answers to "who handles a file". The roster is now substituted into the routing section and the hand-written copy is gone, so the specs are the only source. It also sits inside the tags rather than after </stopping>, which was the unstructured trailing text the XML restructure existed to remove. <preferences> contradicted itself in adjacent paragraphs: code_agent appeared in both "prefer when the question answers itself" and "only when external information is needed". Rewritten as one destination per condition. Documentation: eighteen commits changed eight defaults and touched zero lines of docs, so configuration.md described a setup that had not existed for a day - a retired Groq model, iteration caps of 3, a 180s timeout, and no mention of Anthropic, the effort settings, the token caps or the dollar ceilings. Updated, and a parametrised test now reads the markdown tables and compares each documented default against Settings. Prose does not track code by intention; this makes it fail instead. The comparison is numeric rather than textual, after the first version tripped over "5.00" against 5.0 - a normalisation bug in a test written to guard against the consequences of a normalisation bug.
Measured across every available model on the same input - the benchmark task
written backwards, and "What is the capital of France?" under the same
obfuscation as a control:
haiku-4-5 ANSWERED both
sonnet-4-6 refused both (category: bio)
sonnet-5 refused both (category: general_harms)
opus-5 refused both (category: bio)
The encoding alone triggers it; content is irrelevant, and two models classify a
question about a European capital as a biological risk. Effort made no
difference at either extreme, consistent with every refusal reporting zero
output and zero reasoning tokens: the decision precedes generation, so no
generation parameter can reach it.
Anthropic's server-side `fallbacks` parameter is the documented remedy and is
not supported on claude-sonnet-5 - an Opus/Fable-tier feature. A client-side
retry needs no parameter: on a decline, re-issue that one call to a model that
accepts the input. Haiku is both the model that does and the cheapest, and it
handles one call per refused task rather than any share of the workload.
Wired at the two places that are actually declined. Routing a refused task to a
specialist recovered nothing because the specialist is handed the same text and
declined identically - so the retry lives in both _supervise and the
specialist's reason, with the blind route kept as the last resort when the retry
is disabled or fails.
_route_with takes a model name rather than a client, because constructing one
can raise and an argument is evaluated before the call meant to guard it - the
first version let a missing-credentials error escape past its own try block.
refusal_category moved to core.conversation, where the other response-shape
rules live, so the specialist can use it without importing from the graph.
Also fixes MAX_SPECIALIST_TOKENS, which was documented and defaulted but never
read by load_settings, so the environment variable did nothing.
Three call sites are handed the task text - the router, the specialist and the finalizer - and the classifier declines all three. The retry was wired into the first two. The cost was exact: on 2d83110e the specialist was declined, retried on haiku, reversed the text and answered "right"; the router was declined, retried, and recorded "The code_agent successfully reversed the text and provided the correct answer: right. The task is complete." Then the finalizer was declined, returned an empty reply, and the task was recorded as an error. A solved task lost at the last step because the enumeration stopped one short. A refusal is not an exception, so the existing except could not see it: the reply arrives as a valid message with empty content, which reads downstream as "the model had nothing to say".
wimaan3
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
23 commits. Every change is grounded in a measurement, and several reverse an
earlier one in this same branch — those are called out rather than hidden.
Result: 5/5 on the smoke set, $0.245, every task in 2 supervisor steps.
The instrument was broken
eval/scorers.normalizestripped punctuation before deciding whether a valuewas a number, so it deleted decimal points and minus signs:
Two of the 53 level-1 reference answers are decimals and fifteen are integers.
Every score reported before this commit came through it.
Making a run possible at all
tokens/task that allows 5.7 tasks, so 6/20 was arithmetically unreachable.
conversation.py— three message-shape rules the provider enforces andstubs never did: merged system messages, no trailing assistant turn, and tool
calls paired with their results. Each fixed a defect that OpenAI-compatible
providers accepted while returning garbage.
attachment loader already reads. Grading is now free and instant.
Making failure visible
a
NO_ANSWERsentinel the harness records as an error.the supervisor could not tell research from invention and re-verified
everything: 34,185 tokens on a task solved in round one.
were measured on the same input; only
claude-haiku-4-5accepts it, so adeclined call is retried there. Two of the four classify "What is the capital
of France?" written backwards as a biological risk.
Making it affordable
and the thought producing it each cost one — six turns bought five tools and
nothing to report with. That single miscount caused four commits of workaround.
spreadsheet survives.
Making the next bug findable
ToolCallingLLMemits real tool calls and rejects contract violations likea 400. Every previous stub modelled a cooperative provider, which is why the
same bug shipped twice.
green suite. Every bug here but one was a wiring bug.
run_id,recorded_at,effortandcost_usd, so two runs canbe told apart.
docs/configuration.mdand compares every documented defaultagainst
Settings.Test plan
mypy --strict,ruffcleanHF_SPACEset (the sync workflow has beenskipping silently since August 7) and Space secrets for Tavily and E2B
Known gaps
AnswerCache.saveis not crash-safe🤖 Generated with Claude Code