Report the live eval as eight runs, not one - #1
Merged
Conversation
The first live run scored tool-selection F1 0.88. The fifth scored 0.58. Nothing changed between them. At ten scenarios the run-to-run standard deviation is 0.11 on F1 and 0.14 on task-success, which is wider than the effect being measured, so a single pass is a sample and not a result. So the writeup now reports a mean, a spread and an N. What survives the variance: tool selection genuinely improves on the baseline, seven runs of eight beat it. Grounding is 1.000 with zero deviation across eighty scenario-runs. Task-success does not move and escalation accuracy gets worse, which moves the diagnosis from "every failure is tool selection" to "tool selection is largely solved and escalation is the bottleneck". Nine of those eighty scenario-runs never reached the agent at all: the model emitted a call to a tool that does not exist and the provider rejected the request outright. Five different scenarios, six of the eight runs. It scores as an agent failure while saying nothing about the agent. OPENAI_BASE_URL points the tested OpenAI adapter at any compatible endpoint, which is how these runs reached Groq. Unset, nothing changes. Recorded in the limitations: Gemini 3.x rejects replayed function calls carrying no thought_signature, so multi-turn tool calling currently fails on every available Gemini model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first live run of the eval suite scored tool-selection F1 0.88. The fifth scored 0.58. Same model, same scenarios, same config, minutes apart.
At 10 scenarios the run-to-run sd is 0.106 on F1 and 0.139 on task-success, which is wider than the effect being measured. A single pass is a sample, not a result.
What the writeup now reports
What survives the spread: tool selection genuinely improves (7 of 8 runs beat the baseline, +0.18 on the mean), and grounding is 1.000 with zero deviation across 80 scenario-runs. Task-success is flat and escalation accuracy is worse, which moves the diagnosis from "every failure is tool selection" to "tool selection is largely solved, escalation is the bottleneck".
11% of scenario-runs never reached the agent
9 of 80 died on a provider-side 400: the model called
response_to_customer, which does not exist in this repo, and the API rejected the whole request. Five different scenarios, six of eight runs. It scores as an agent failure while saying nothing about the agent. Two consecutive runs happened to hit the same scenario, which is just long enough to mistake it for a reproducible bug.Code
OPENAI_BASE_URLconfig seam points the already-tested OpenAI adapter at any compatible endpoint (Groq, Together, OpenRouter, local). Unset, behaviour is unchanged. This is how the runs reached Groq.Also recorded under limitations: Gemini 3.x rejects replayed function calls with no
thought_signature, which the adapter does not emit, so multi-turn tool calling fails on every currently available Gemini model.gemini-2.0-flash, which predates the requirement, is retired.33 tests pass.