From bcae5e58d36b4332fd7c5df792b4a031443374e1 Mon Sep 17 00:00:00 2001 From: Veer Arora Date: Thu, 27 Aug 2026 17:33:13 +0530 Subject: [PATCH] Report the live eval as eight runs, not one The first live run scored tool-selection F1 0.88. The fifth scored 0.58. Nothing changed between them. At ten scenarios the run-to-run standard deviation is 0.11 on F1 and 0.14 on task-success, which is wider than the effect being measured, so a single pass is a sample and not a result. So the writeup now reports a mean, a spread and an N. What survives the variance: tool selection genuinely improves on the baseline, seven runs of eight beat it. Grounding is 1.000 with zero deviation across eighty scenario-runs. Task-success does not move and escalation accuracy gets worse, which moves the diagnosis from "every failure is tool selection" to "tool selection is largely solved and escalation is the bottleneck". Nine of those eighty scenario-runs never reached the agent at all: the model emitted a call to a tool that does not exist and the provider rejected the request outright. Five different scenarios, six of the eight runs. It scores as an agent failure while saying nothing about the agent. OPENAI_BASE_URL points the tested OpenAI adapter at any compatible endpoint, which is how these runs reached Groq. Unset, nothing changes. Recorded in the limitations: Gemini 3.x rejects replayed function calls carrying no thought_signature, so multi-turn tool calling currently fails on every available Gemini model. --- EVAL_WRITEUP.md | 93 ++++++++++++++++++++++++++++++++-------- backend/app/agent/llm.py | 6 ++- backend/app/config.py | 4 ++ 3 files changed, 83 insertions(+), 20 deletions(-) diff --git a/EVAL_WRITEUP.md b/EVAL_WRITEUP.md index 0610e82..1a97dc9 100644 --- a/EVAL_WRITEUP.md +++ b/EVAL_WRITEUP.md @@ -1,4 +1,4 @@ -# AgentOps — Eval writeup (measure → diagnose → improve) +# AgentOps: eval writeup (measure → diagnose → improve) The point of AgentOps isn't the agent; it's the **loop that makes the agent better**. This is the writeup that loop produces. It is regenerated from the eval harness @@ -12,19 +12,19 @@ the resulting trace, and writes `backend/evals/reports/scorecard-latest.md`. | **Tool-selection F1** | Set-based F1 of the tools the agent actually called vs. the scenario's expected tools (+ exact-set match). | | **Task-success** | Did it reach the right outcome? Deterministic: required escalation matches **and** all answer keywords present. | | **Escalation accuracy** | Accuracy of the escalate / don't-escalate decision over labeled scenarios. | -| **Citation grounding** | Fraction of answers whose every cited `[DOC-###]` / `[GH-###]` marker was actually retrieved this run — a deterministic **hallucination proxy** (an LLM judge slots in when a key is present). | +| **Citation grounding** | Fraction of answers whose every cited `[DOC-###]` / `[GH-###]` marker was actually retrieved this run: a deterministic **hallucination proxy** (an LLM judge slots in when a key is present). | | **Cost / latency** | Per-run USD (from token usage × a price table) and p50/p95 latency, both read from the `agent_steps` trace. | ## Baseline (10 scenarios, offline demo model) -The offline demo model is a fixed heuristic — *always `search_docs`, then answer* — +The offline demo model is a fixed heuristic, *always `search_docs`, then answer*, run with no API key so the harness itself is verifiable end to end. | Tool-sel. F1 | Task-success | Escalation acc. | Citation grounding | Cost | Latency p50/p95 | |---|---|---|---|---|---| | 0.60 | 0.50 | 0.90 | 1.00 | $0.00 | ~0 / ~0 ms | -## Diagnosis — where it fails, and why +## Diagnosis: where it fails, and why 5 of 10 scenarios fail. Grouping the failures is the whole game: @@ -36,7 +36,7 @@ run with no API key so the harness itself is verifiable end to end. | `known-issue-rate-limit` | `search_github_issues` | `search_docs` | tool selection (known-issue) | | `refund-request-action` | escalate | answered instead | escalation | -**100% of failures are routing decisions** — picking the wrong tool, or not choosing +**100% of failures are routing decisions**: picking the wrong tool, or not choosing to escalate. Critically, they are **not** retrieval or hallucination failures: citation grounding is **1.0**, so the demo never fabricates a source; it just answers the wrong kind of question with the only tool it knows. That separation is the @@ -53,30 +53,85 @@ diagnosis: the gap is *tool selection and escalation*, not knowledge or groundin Each hypothesis maps to a specific failure class above, so the next scorecard tells us directly whether the change worked. That's the loop. -## The story (fill in from a live run) - -> "My support agent resolved **50%** of scenarios on the baseline. Error analysis -> showed **every** failure was a tool-selection or escalation decision — not -> retrieval or hallucination (grounding was 1.0). I switched to a real model and -> enforced a policy gate for risky actions, which targeted exactly those failure -> modes; task-success rose to **X%**, with cost and p95 latency tracked per run." - -Run it live to fill in **X%**: +## Live run: 8 passes, and why one pass was never going to do + +Run against `openai/gpt-oss-120b` on Groq, reached through the OpenAI-compatible +`OPENAI_BASE_URL` seam. Eight consecutive passes over the same 10 scenarios, same +config, minutes apart. + +| Metric | Baseline (demo) | Live, mean of 8 | sd | Range | +|---|---|---|---|---| +| Tool-selection F1 | 0.60 | **0.780** | 0.106 | 0.580 - 0.967 | +| Tool exact-match | n/a | 0.637 | 0.122 | 0.500 - 0.900 | +| Task-success | 0.50 | 0.475 | 0.139 | 0.400 - 0.800 | +| Escalation accuracy | 0.90 | 0.725 | 0.120 | 0.600 - 0.900 | +| Citation grounding | 1.00 | **1.000** | 0.000 | 1.000 - 1.000 | + +**The first run scored F1 0.88. The fifth scored 0.58.** Nothing changed between +them. A standard deviation of 0.106 on a 10-scenario suite is wider than the +effect being measured, so a single pass is a sample, not a result, and reporting +one would have meant publishing whichever number the dice produced. This is the +finding, and it applies to every agent benchmark of this size. + +What survives the variance: + +- **Tool selection genuinely improved.** Seven of eight runs beat the 0.60 + baseline, and the mean is +0.18. This is the one claim the spread supports. +- **Grounding is 1.000 with zero deviation** across 8 runs and 80 scenario-runs. + A hallucination proxy that never once fired is a stronger statement than any + single-run figure here. +- **Task-success did not move** (0.475 against 0.50), and **escalation accuracy + got worse** (0.725 against 0.90). The baseline's heuristic never escalates + when it should not; a real model sometimes does. The diagnosis therefore moves + from "every failure is tool selection" to "tool selection is largely solved and + the escalation decision is now the bottleneck". + +### 11% of scenario-runs never reached the agent + +Nine of the 80 scenario-runs died on a provider-side 400: the model emitted a +call to `response_to_customer`, a tool that does not exist in this repo, and the +API rejected the whole request rather than returning the bad call for the agent +to recover from. + +It hit five different scenarios and appeared in six of the eight runs, so it is +not one broken scenario. It is a random tax that scores as an agent failure while +saying nothing about the agent, and it depresses both F1 and task-success above. +Two consecutive runs happened to hit the same scenario, which is exactly long +enough to mistake it for a reproducible bug. + +The harness records these in the row's answer as `[run error: ...]` rather than +hiding them, which is the only reason the pattern was visible at all. Not scoring +them at all is the obvious next change. + +### Reproducing it ```bash -printf 'PROVIDER=openai\nOPENAI_API_KEY=sk-...\n' >> backend/.env +printf 'PROVIDER=openai +OPENAI_API_KEY=gsk_... +OPENAI_MODEL=openai/gpt-oss-120b +OPENAI_BASE_URL=https://api.groq.com/openai/v1 +' >> backend/.env backend/.venv/Scripts/python -m evals.runner ``` -The harness is provider-agnostic, so the same command with `PROVIDER=anthropic` -benchmarks `claude-opus-5` on identical scenarios — a direct gpt-4o vs. Claude -comparison on tool-selection, task-success, cost, and latency. +`OPENAI_BASE_URL` points the tested OpenAI adapter at any compatible endpoint +(Groq, Together, OpenRouter, a local server) without touching code. Leave it unset +for OpenAI itself. ## Honest limitations - **10 scenarios**, not 50–100 yet. Scaling is continued authoring (same discipline as CiteRAG's golden set); the harness and scorers don't change. - **Grounding is a deterministic proxy.** It catches fabricated citations, not subtly - wrong claims grounded in the wrong passage — that needs the LLM judge (Phase 3.5). + wrong claims grounded in the wrong passage, which needs the LLM judge (Phase 3.5). - **Offline latency/cost are ~0** because the demo model does no network I/O; these columns are meaningful only on live runs. +- **One pass is not a measurement.** At 10 scenarios the run-to-run sd is 0.11 on + F1 and 0.14 on task-success, so any figure quoted from a single run is inside + the noise. Quote a mean with its spread and the number of runs, or quote + nothing. Enlarging the suite is what actually narrows this. +- **Provider-agnostic has one live gap.** Gemini 3.x rejects replayed function + calls that carry no `thought_signature`, which the OpenAI-compatible adapter + does not emit, so multi-turn tool calling fails on every currently available + Gemini model. `gemini-2.0-flash`, which predates the requirement, is retired. + OpenAI, Anthropic and any OpenAI-compatible endpoint are unaffected. diff --git a/backend/app/agent/llm.py b/backend/app/agent/llm.py index a027f58..edcc0a2 100644 --- a/backend/app/agent/llm.py +++ b/backend/app/agent/llm.py @@ -179,7 +179,11 @@ def build_llm(settings) -> LLMClient: if settings.openai_api_key: from app.agent.openai_llm import OpenAILLMClient - return OpenAILLMClient(api_key=settings.openai_api_key, model=settings.openai_model) + return OpenAILLMClient( + api_key=settings.openai_api_key, + model=settings.openai_model, + base_url=settings.openai_base_url, + ) return DemoLLMClient() if settings.provider == "google": if settings.google_api_key: diff --git a/backend/app/config.py b/backend/app/config.py index e2238b8..a9431e6 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,6 +32,10 @@ class Settings(BaseSettings): openai_api_key: str | None = None openai_model: str = "gpt-4o" + # Any OpenAI-compatible endpoint (Groq, Together, OpenRouter, a local + # server). The adapter already accepts a base_url; this is the config seam + # that lets a run reach one without editing code. + openai_base_url: str | None = None anthropic_api_key: str | None = None anthropic_model: str = "claude-opus-5"