diff --git a/EVAL_WRITEUP.md b/EVAL_WRITEUP.md index 0610e82..1a97dc9 100644 --- a/EVAL_WRITEUP.md +++ b/EVAL_WRITEUP.md @@ -1,4 +1,4 @@ -# AgentOps — Eval writeup (measure → diagnose → improve) +# AgentOps: eval writeup (measure → diagnose → improve) The point of AgentOps isn't the agent; it's the **loop that makes the agent better**. This is the writeup that loop produces. It is regenerated from the eval harness @@ -12,19 +12,19 @@ the resulting trace, and writes `backend/evals/reports/scorecard-latest.md`. | **Tool-selection F1** | Set-based F1 of the tools the agent actually called vs. the scenario's expected tools (+ exact-set match). | | **Task-success** | Did it reach the right outcome? Deterministic: required escalation matches **and** all answer keywords present. | | **Escalation accuracy** | Accuracy of the escalate / don't-escalate decision over labeled scenarios. | -| **Citation grounding** | Fraction of answers whose every cited `[DOC-###]` / `[GH-###]` marker was actually retrieved this run — a deterministic **hallucination proxy** (an LLM judge slots in when a key is present). | +| **Citation grounding** | Fraction of answers whose every cited `[DOC-###]` / `[GH-###]` marker was actually retrieved this run: a deterministic **hallucination proxy** (an LLM judge slots in when a key is present). | | **Cost / latency** | Per-run USD (from token usage × a price table) and p50/p95 latency, both read from the `agent_steps` trace. | ## Baseline (10 scenarios, offline demo model) -The offline demo model is a fixed heuristic — *always `search_docs`, then answer* — +The offline demo model is a fixed heuristic, *always `search_docs`, then answer*, run with no API key so the harness itself is verifiable end to end. | Tool-sel. F1 | Task-success | Escalation acc. | Citation grounding | Cost | Latency p50/p95 | |---|---|---|---|---|---| | 0.60 | 0.50 | 0.90 | 1.00 | $0.00 | ~0 / ~0 ms | -## Diagnosis — where it fails, and why +## Diagnosis: where it fails, and why 5 of 10 scenarios fail. Grouping the failures is the whole game: @@ -36,7 +36,7 @@ run with no API key so the harness itself is verifiable end to end. | `known-issue-rate-limit` | `search_github_issues` | `search_docs` | tool selection (known-issue) | | `refund-request-action` | escalate | answered instead | escalation | -**100% of failures are routing decisions** — picking the wrong tool, or not choosing +**100% of failures are routing decisions**: picking the wrong tool, or not choosing to escalate. Critically, they are **not** retrieval or hallucination failures: citation grounding is **1.0**, so the demo never fabricates a source; it just answers the wrong kind of question with the only tool it knows. That separation is the @@ -53,30 +53,85 @@ diagnosis: the gap is *tool selection and escalation*, not knowledge or groundin Each hypothesis maps to a specific failure class above, so the next scorecard tells us directly whether the change worked. That's the loop. -## The story (fill in from a live run) - -> "My support agent resolved **50%** of scenarios on the baseline. Error analysis -> showed **every** failure was a tool-selection or escalation decision — not -> retrieval or hallucination (grounding was 1.0). I switched to a real model and -> enforced a policy gate for risky actions, which targeted exactly those failure -> modes; task-success rose to **X%**, with cost and p95 latency tracked per run." - -Run it live to fill in **X%**: +## Live run: 8 passes, and why one pass was never going to do + +Run against `openai/gpt-oss-120b` on Groq, reached through the OpenAI-compatible +`OPENAI_BASE_URL` seam. Eight consecutive passes over the same 10 scenarios, same +config, minutes apart. + +| Metric | Baseline (demo) | Live, mean of 8 | sd | Range | +|---|---|---|---|---| +| Tool-selection F1 | 0.60 | **0.780** | 0.106 | 0.580 - 0.967 | +| Tool exact-match | n/a | 0.637 | 0.122 | 0.500 - 0.900 | +| Task-success | 0.50 | 0.475 | 0.139 | 0.400 - 0.800 | +| Escalation accuracy | 0.90 | 0.725 | 0.120 | 0.600 - 0.900 | +| Citation grounding | 1.00 | **1.000** | 0.000 | 1.000 - 1.000 | + +**The first run scored F1 0.88. The fifth scored 0.58.** Nothing changed between +them. A standard deviation of 0.106 on a 10-scenario suite is wider than the +effect being measured, so a single pass is a sample, not a result, and reporting +one would have meant publishing whichever number the dice produced. This is the +finding, and it applies to every agent benchmark of this size. + +What survives the variance: + +- **Tool selection genuinely improved.** Seven of eight runs beat the 0.60 + baseline, and the mean is +0.18. This is the one claim the spread supports. +- **Grounding is 1.000 with zero deviation** across 8 runs and 80 scenario-runs. + A hallucination proxy that never once fired is a stronger statement than any + single-run figure here. +- **Task-success did not move** (0.475 against 0.50), and **escalation accuracy + got worse** (0.725 against 0.90). The baseline's heuristic never escalates + when it should not; a real model sometimes does. The diagnosis therefore moves + from "every failure is tool selection" to "tool selection is largely solved and + the escalation decision is now the bottleneck". + +### 11% of scenario-runs never reached the agent + +Nine of the 80 scenario-runs died on a provider-side 400: the model emitted a +call to `response_to_customer`, a tool that does not exist in this repo, and the +API rejected the whole request rather than returning the bad call for the agent +to recover from. + +It hit five different scenarios and appeared in six of the eight runs, so it is +not one broken scenario. It is a random tax that scores as an agent failure while +saying nothing about the agent, and it depresses both F1 and task-success above. +Two consecutive runs happened to hit the same scenario, which is exactly long +enough to mistake it for a reproducible bug. + +The harness records these in the row's answer as `[run error: ...]` rather than +hiding them, which is the only reason the pattern was visible at all. Not scoring +them at all is the obvious next change. + +### Reproducing it ```bash -printf 'PROVIDER=openai\nOPENAI_API_KEY=sk-...\n' >> backend/.env +printf 'PROVIDER=openai +OPENAI_API_KEY=gsk_... +OPENAI_MODEL=openai/gpt-oss-120b +OPENAI_BASE_URL=https://api.groq.com/openai/v1 +' >> backend/.env backend/.venv/Scripts/python -m evals.runner ``` -The harness is provider-agnostic, so the same command with `PROVIDER=anthropic` -benchmarks `claude-opus-5` on identical scenarios — a direct gpt-4o vs. Claude -comparison on tool-selection, task-success, cost, and latency. +`OPENAI_BASE_URL` points the tested OpenAI adapter at any compatible endpoint +(Groq, Together, OpenRouter, a local server) without touching code. Leave it unset +for OpenAI itself. ## Honest limitations - **10 scenarios**, not 50–100 yet. Scaling is continued authoring (same discipline as CiteRAG's golden set); the harness and scorers don't change. - **Grounding is a deterministic proxy.** It catches fabricated citations, not subtly - wrong claims grounded in the wrong passage — that needs the LLM judge (Phase 3.5). + wrong claims grounded in the wrong passage, which needs the LLM judge (Phase 3.5). - **Offline latency/cost are ~0** because the demo model does no network I/O; these columns are meaningful only on live runs. +- **One pass is not a measurement.** At 10 scenarios the run-to-run sd is 0.11 on + F1 and 0.14 on task-success, so any figure quoted from a single run is inside + the noise. Quote a mean with its spread and the number of runs, or quote + nothing. Enlarging the suite is what actually narrows this. +- **Provider-agnostic has one live gap.** Gemini 3.x rejects replayed function + calls that carry no `thought_signature`, which the OpenAI-compatible adapter + does not emit, so multi-turn tool calling fails on every currently available + Gemini model. `gemini-2.0-flash`, which predates the requirement, is retired. + OpenAI, Anthropic and any OpenAI-compatible endpoint are unaffected. diff --git a/backend/app/agent/llm.py b/backend/app/agent/llm.py index a027f58..edcc0a2 100644 --- a/backend/app/agent/llm.py +++ b/backend/app/agent/llm.py @@ -179,7 +179,11 @@ def build_llm(settings) -> LLMClient: if settings.openai_api_key: from app.agent.openai_llm import OpenAILLMClient - return OpenAILLMClient(api_key=settings.openai_api_key, model=settings.openai_model) + return OpenAILLMClient( + api_key=settings.openai_api_key, + model=settings.openai_model, + base_url=settings.openai_base_url, + ) return DemoLLMClient() if settings.provider == "google": if settings.google_api_key: diff --git a/backend/app/config.py b/backend/app/config.py index e2238b8..a9431e6 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,6 +32,10 @@ class Settings(BaseSettings): openai_api_key: str | None = None openai_model: str = "gpt-4o" + # Any OpenAI-compatible endpoint (Groq, Together, OpenRouter, a local + # server). The adapter already accepts a base_url; this is the config seam + # that lets a run reach one without editing code. + openai_base_url: str | None = None anthropic_api_key: str | None = None anthropic_model: str = "claude-opus-5"