feat(sdk): add per-call prompt token composition metrics

[georgeglarson]

HUMAN:

Validated. Human thinks this is correct also.

---

AGENT:

Verification beyond unit tests: the default agent (get_default_agent, 19 tools) ran live on
tasks from the published harness-benchmark short suite
(https://github.com/rajshah4/harness-benchmark) on two model lanes: MiniMax-M3 (three tasks, 41
calls; litellm has no tokenizer mapping for this model, so estimates use the fallback
tokenizer) and gpt-4o-mini (two tasks, 37 calls; litellm maps this model to its real o200k
tokenizer). Every call on both lanes recorded both a PromptComposition and a provider usage
record, with zero counting failures. The table under "Why" comes from those runs; per-call data
is attached under .pr/evidence/ (PR-only reviewer context, removed after approval per repo
policy).

Re-verified after rebasing onto current upstream/main: p09-task-01 on gpt-4o-mini, 20 calls, verifier PASS, est/provider ratio 0.99-1.01.

Estimator validation: the per-call ratio of the estimated component sum to provider-reported
prompt_tokens runs 0.97-1.00 on the mapped-tokenizer lane (median 1.00 on the trivial task)
and 0.85-0.97 on the unmapped lane. The residual gap is consistent with request framing and
tokenizer mapping differences; the component split is the finding, not the absolute sum.

Commands run on the branch:

• uv run pytest tests/sdk/llm/test_prompt_composition.py -q → 20 passed
• uv run pytest tests/sdk/llm -q → 999 passed
• uv run pytest tests/sdk -q → 5970 passed, 17 failed (pre-existing on clean upstream/main in this environment; the failure list is byte-identical with and without this branch), 7 skipped, 10 xfailed
• uv run pre-commit run --files <each changed file> → all hooks pass

Why

The SDK cannot say what a prompt is made of. Metrics records provider totals, but nothing
decomposes a call into system prompt, tool schemas, history, and the latest message. Two
consumers need this decomposition: the context-window UI work (the usage panel renders totals,
with no breakdown available to display), and tool-loading studies such as #4083, where the
per-call cost of the tool surface is the quantity under discussion.

Measured on the default agent (live runs, MiniMax-M3 lane, litellm token_counter):

task	calls	avg system	avg tool schemas	avg history	tool schemas re-sent, total
trivial rename	5	3,331	5,749	533	28,745
medium refactor	26	3,331	5,749	10,462	149,474
cache implementation	10	3,331	5,749	2,605	57,490

The gpt-4o-mini lane (mapped tokenizer) reproduces the same shape: system ≈ 3,340 and tool
schemas ≈ 5,702 estimated tokens per call, constant across 37 calls.

The preamble (system + tool schemas ≈ 9,080 estimated tokens) is constant per call: ~79% of the
average call on the trivial task (average provider-reported input 11,478 tokens), and ~84% of
first calls. On the 26-call task, re-sent tool schemas came to 27% of the run's total
provider-reported input tokens.

Summary

• New PromptComposition per-call record on Metrics: system / tool / history / latest-message
  tokens, an is_estimate flag, and a response_id joining to the authoritative TokenUsage.
• Computed at the LLM completion/responses boundary with litellm's token_counter, opt-in
  via enable_prompt_composition=True (default off: no tokenization pass runs, no records are
  appended). When enabled, cost scales linearly with prompt size (~10-20 ms on typical agent
  payloads, ~31 ms measured at ~100K tokens); best-effort (counting failure records nothing,
  the call is never affected).
• Additive-only, so older persisted Metrics payloads still load. Exported as
  openhands.sdk.PromptComposition. Agent steps that send tools as native function-calling
  schemas have tool_tokens > 0; auxiliary calls that pass no tools (condenser, title
  generation) have tool_tokens == 0.

Issue Number

Related to #4083 (this PR provides the measurement; it does not implement deferred loading).

How to Test

uv run pytest tests/sdk/llm/test_prompt_composition.py -q

For a live check: build an LLM, make any completion, then inspect
llm.metrics.latest_prompt_composition.

Video/Screenshots

Type

• Feature

Notes

Companion docs PR: OpenHands/docs#757

[github-actions]

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. Because this is a fork PR, the directory will be automatically removed from main immediately after merge.

[rajshah4]

Really nice work! Thanks for validating it against real agent runs. The decomposition is exactly the kind of measurement we need for tool-loading and context-budget studies.

I'm supportive of getting this into the SDK. One concern on the current shape: counting is always on for every LLM call, for every user. The measured cost is small (~10-20ms typical, ~31ms at 100K tokens), but it's a global tax on a metric most users won't consume.

Could we put this behind an opt-in flag (something like enable_prompt_composition on the LLM, defaulting off) so it's available when you want to troubleshoot a prompt or run a tool-loading study, without imposing the extra tokenization pass on every default user?

Drafted with help from an AI agent (OpenHands) on behalf of @rajshah4.

[georgeglarson]

Done in 4a9c39df: counting is now opt-in via enable_prompt_composition on the LLM, default
off. When off, no tokenization pass runs and no records are appended; a spy test asserts zero
token_counter calls on that path. When on, behavior is unchanged from the numbers above.
Companion docs PR updated too. Thanks for the review.

[VascoSch92]

Thanks for the PR. I looked a little inside.

it is actually adding a little of confusion and repetition.

Lets take an example

"composition": {
    "model": "gpt-4o-mini", 
    "system_prompt_tokens": 3340, 
    "tool_tokens": 5702, 
    "history_tokens": 0, 
    "latest_message_tokens": 38, 
    "is_estimate": true, 
    "response_id": "chatcmpl-EGVEG8NSZIEywDIyNQnqt99X5R4tJ"}

• model: you have already this information. So this is just a repetition.
• system prompt tokens never change from the beginning of the convo. But this is saving them every turn right?
• what it means tool_tokens? tool output? tool call? tool description? If it is the third: these are already in the system prompt (so is a subset)
• what history_tokens? Cached ones? Are the one of the previous round? But actually you don't need to save this information because you can aways do the difference between two saved composite metrics
• why it is estimated? I mean, this is just a computation and we have the correct data from the provider right?
• response id is a repetition. We do have already this info.

[rajshah4]

Thanks for taking a look, @VascoSch92 — these are fair questions to raise on a new metrics surface. A few of them are worth revisiting against the evidence traces attached to the PR (.pr/evidence/), since the live runs clarify the design choices. Using the gpt-4o-mini task-01 lane as a reference point:

• seq 0: provider reports prompt_tokens=9172 (a single total), while the composition records system=3340, tool=5702, history=0, latest=38.

On (5) why it's an estimate — this is the crux. The provider returns one aggregate number, not a breakdown. There's no provider API that returns the 3340/5702/0/38 split; that decomposition only exists client-side. Producing that breakdown is the entire reason for the feature — if the provider gave it to us, we wouldn't need this PR. The is_estimate flag just makes the provenance explicit so nobody mistakes the client-side split for a provider-reported figure.

On (3) tool_tokens — on native-function-calling models the traces show system=3,340 and tool=5,702 as two independent, constant buckets. If the tool schemas were folded into the system prompt, system would read ~9,080 and tool would be 0 — it doesn't. They're sent as the separate tools request parameter. For the non-native path where schemas are rendered into the prompt, the PR already passes tools=None so tool_tokens=0 and there's no double count (covered by test_mock_tools_does_not_double_count_tool_schemas). Your naming point is fair though — tool_schema_tokens would be clearer than tool_tokens and would remove the ambiguity about whether it means schemas, calls, or outputs.

On (4) history derivable by diffing — not quite cleanly. seq 0 → seq 1: history goes 0 → 73, but latest_0=38. The gap is the assistant's completion (38 tokens, which lives in TokenUsage, not in any composition bucket) plus framing overhead. Reconstructing history would require a cross-list join against token_usages.completion_tokens and still only approximate the value. Storing it directly is exact, and the cost is a handful of ints per call.

On (2) system constant per turn — true in these traces (3,340 across all 9 calls of task-01), and a reasonable thing to flag. The per-call shape mirrors the existing TokenUsage / ResponseLatency records though, and it lets a caller detect a mid-conversation system change (skill activation, condenser rewrite, dynamic system message) without needing a special event type. Happy to consider whether a "preamble summary" record is worth adding, but the per-call cost is small.

On (1) and (6) model / response_id — both match the existing pattern: TokenUsage carries model, and ResponseLatency carries both model and response_id. Keeping PromptComposition consistent lets each record stand on its own. response_id specifically is the join key back to token_usages, and the docs note the two lists can diverge (skipped composition, or a response with no usage), so positional joining isn't safe.

The one genuinely actionable nit I see is the tool_tokens naming — tool_schema_tokens would be clearer. The rest looks consistent with the existing metrics design, and the traces back up the choices. Appreciate the review.

Drafted with help from an AI agent (OpenHands) on behalf of @rajshah4.

[VascoSch92]

On (1) and (6) model / response_id — both match the existing pattern: TokenUsage carries model, and ResponseLatency carries both model and response_id. Keeping PromptComposition consistent lets each record stand on its own. response_id specifically is the join key back to token_usages, and the docs note the two lists can diverge (skipped composition, or a response with no usage), so positional joining isn't safe.

If old code is not optimal doesn't mean that we should continue to create not optimal code.

On (5) why it's an estimate — this is the crux. The provider returns one aggregate number, not a breakdown. There's no provider API that returns the 3340/5702/0/38 split; that decomposition only exists client-side. Producing that breakdown is the entire reason for the feature — if the provider gave it to us, we wouldn't need this PR. The is_estimate flag just makes the provenance explicit so nobody mistakes the client-side split for a provider-reported figure.

It is an estimate if you can not compute that correctly. But in this case we can. So there is no motivation to have estimate here.

I think all that can be resume in a script that you can run after an agent conversation to extract the information. but at run time we don't need that.

[rajshah4]

Fair point on the architecture — I think you're right that starting with a script is the better first step. Get the decomposition into people's hands as an offline tool, validate it's useful on real runs, and earn the runtime API surface before committing to it. No disagreement from me on that.

@georgeglarson — the decomposition logic you've built here (compute_prompt_composition, the tool-token marginal delta, the Responses payload conversion) is solid and reusable as-is for a script that ingests a conversation's logged messages + token_usages and emits the per-call breakdown. The hard part is done. Would you be open to repackaging this as a post-run analysis tool rather than a runtime metric? The test coverage carries over too. One addition worth considering: maybe we can include some visualization tools alongside the script to help people understand what's going on — a simple breakdown of where tokens go per call (system / tool schemas / history / latest), maybe a per-call trend over the conversation, would make the output much more approachable than raw JSON. Even a basic matplotlib or text-based chart would go a long way for troubleshooting.

If the script proves the value and there's later appetite for live observability (context-window UI, real-time tool-loading studies), the runtime path is already designed and can be revisited then. But starting with the script is the lower-risk way in.

Happy to run the script against the harness-benchmark suite and report back on what it surfaces.

Drafted with help from an AI agent (OpenHands) on behalf of @rajshah4.