From 991a59cf66ef7f52140f89ef5cd9f0412a59ad67 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 14:53:22 -0400 Subject: [PATCH 1/4] docs(sdk): document per-call prompt token composition metrics Companion to the PromptComposition feature on software-agent-sdk feat/llm-prompt-composition-metrics: per-call decomposition of prompt tokens into system prompt, tool schemas, conversation history, and latest message, recorded in LLM Metrics. Co-authored-by: openhands --- docs.json | 3 +- sdk/guides/llm-prompt-composition.mdx | 71 +++++++++++++++++++++++++++ sdk/guides/metrics.mdx | 2 + 3 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 sdk/guides/llm-prompt-composition.mdx diff --git a/docs.json b/docs.json index 41479408b..d06686b2e 100644 --- a/docs.json +++ b/docs.json @@ -361,7 +361,8 @@ "sdk/guides/llm-image-input", "sdk/guides/llm-error-handling", "sdk/guides/llm-fallback", - "sdk/guides/llm-profile-store" + "sdk/guides/llm-profile-store", + "sdk/guides/llm-prompt-composition" ] }, { diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx new file mode 100644 index 000000000..11562daf2 --- /dev/null +++ b/sdk/guides/llm-prompt-composition.mdx @@ -0,0 +1,71 @@ +--- +title: Prompt Token Composition +description: Break down prompt tokens per LLM call into system prompt, tool schemas, conversation history, and the latest message. +--- + +## Overview + +Every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: + +- `system_prompt_tokens` - Tokens in system messages +- `tool_tokens` - Tokens in the tool schemas included with the call +- `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) +- `latest_message_tokens` - Tokens in the latest observation or user message + +The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. + +## Accessing the Composition + +Each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition`: + +```python icon="python" +conversation.run() + +composition = llm.metrics.latest_prompt_composition +assert composition is not None +print(f"System prompt: {composition.system_prompt_tokens}") +print(f"Tool schemas: {composition.tool_tokens}") +print(f"History: {composition.history_tokens}") +print(f"Latest message: {composition.latest_message_tokens}") +``` + +Each record carries the `response_id` of its call, so you can join it with the provider-reported record in `llm.metrics.token_usages`: + +```python icon="python" +for composition, usage in zip( + llm.metrics.prompt_compositions, llm.metrics.token_usages +): + estimated = ( + composition.system_prompt_tokens + + composition.tool_tokens + + composition.history_tokens + + composition.latest_message_tokens + ) + print( + f"{composition.response_id}: estimated {estimated}, " + f"provider reported {usage.prompt_tokens}" + ) +``` + +Agent steps always include the agent's tool list, so calls from the primary agent loop have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. + +## Estimates vs Provider-Reported Usage + +Composition counts are **client-side estimates**, computed with the model's tokenizer before the request is sent. The provider-reported `TokenUsage` remains the authoritative accounting: + +- `is_estimate` is `True` on records produced by the client-side estimator. +- Each component is counted independently, so per-message framing overhead is included in every component and the components do not necessarily sum exactly to the provider-reported `prompt_tokens`. +- When no tokenizer is available for a model, composition recording is skipped for that call; the call itself is unaffected. + + + Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. + + +## Performance + +The estimator adds roughly 10-20 ms per call on a typical agent payload (measured with the default 19-tool agent on a ~27K-token prompt) - negligible next to network latency - so it is always on. + +## Next Steps + +- **[Metrics Tracking](/sdk/guides/metrics)** - Token usage, costs, and latency metrics for your agents +- **[Context Condenser](/sdk/guides/context-condenser)** - How OpenHands keeps history within the context window diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index 023306602..e4035318d 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -41,6 +41,7 @@ The `llm.metrics` object is an instance of the [Metrics class](https://github.co - `costs` - List of individual cost records per API call - `token_usages` - List of detailed token usage records per API call - `response_latencies` - List of response latency metrics per API call +- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message) For more details on the available metrics and methods, refer to the [source code](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-sdk/openhands/sdk/llm/utils/metrics.py). @@ -409,5 +410,6 @@ for usage_id, metrics in conversation.conversation_stats.usage_to_metrics.items( ## Next Steps +- **[Prompt Token Composition](/sdk/guides/llm-prompt-composition)** - Break down prompt tokens per call into system prompt, tool schemas, history, and latest message - **[Context Condenser](/sdk/guides/context-condenser)** - Learn about context management and how it uses separate LLMs - **[LLM Routing](/sdk/guides/llm-routing)** - Optimize costs with smart routing between different models From 99bcd6e354c4d13dbe12ed29fa3a73b5eee1d4d2 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 16:02:33 -0400 Subject: [PATCH 2/4] docs(sdk): fix prompt composition join example and estimator caveats Address review findings: join compositions to token usage by response_id instead of positionally, qualify tool_tokens > 0 as native-FC agent steps (mock-tools renders schemas into prompt text), note litellm's tool serialization convention and fallback tokenizer, and document the linear-in-prompt-size counting cost. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 11562daf2..2203632c2 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -12,7 +12,7 @@ Every LLM call records a per-call decomposition of its prompt tokens, so you can - `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) - `latest_message_tokens` - Tokens in the latest observation or user message -The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. +The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. ## Accessing the Composition @@ -29,12 +29,15 @@ print(f"History: {composition.history_tokens}") print(f"Latest message: {composition.latest_message_tokens}") ``` -Each record carries the `response_id` of its call, so you can join it with the provider-reported record in `llm.metrics.token_usages`: +Each record carries the `response_id` of its call. Join records with the provider-reported usage in `llm.metrics.token_usages` by `response_id` rather than by position - the two lists can diverge when a composition is skipped or a response carries no usage: ```python icon="python" -for composition, usage in zip( - llm.metrics.prompt_compositions, llm.metrics.token_usages -): +usage_by_id = {u.response_id: u for u in llm.metrics.token_usages} + +for composition in llm.metrics.prompt_compositions: + usage = usage_by_id.get(composition.response_id) + if usage is None: + continue estimated = ( composition.system_prompt_tokens + composition.tool_tokens @@ -47,7 +50,11 @@ for composition, usage in zip( ) ``` -Agent steps always include the agent's tool list, so calls from the primary agent loop have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. +Agent steps that send tools as native function-calling schemas have `tool_tokens > 0`. Auxiliary calls that pass no tools - for example the [context condenser](/sdk/guides/context-condenser) or title generation - are recorded with `tool_tokens == 0`. + + + On models without native function calling, the SDK renders tool schemas into the prompt text instead of sending them as tool parameters. Those agent steps are recorded with `tool_tokens == 0` and the schema tokens appear in the message buckets instead. + ## Estimates vs Provider-Reported Usage @@ -55,7 +62,9 @@ Composition counts are **client-side estimates**, computed with the model's toke - `is_estimate` is `True` on records produced by the client-side estimator. - Each component is counted independently, so per-message framing overhead is included in every component and the components do not necessarily sum exactly to the provider-reported `prompt_tokens`. -- When no tokenizer is available for a model, composition recording is skipped for that call; the call itself is unaffected. +- Tool schema counts follow litellm's `token_counter` serialization convention for tools, which can differ from the provider's wire-format tokenization. +- For models litellm has no tokenizer mapping for, counts use litellm's fallback tokenizer and may deviate more from the provider's counts. +- When token counting fails or is disabled (for example `litellm.disable_token_counter`), composition recording is skipped for that call; the call itself is unaffected. Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. @@ -63,7 +72,7 @@ Composition counts are **client-side estimates**, computed with the model's toke ## Performance -The estimator adds roughly 10-20 ms per call on a typical agent payload (measured with the default 19-tool agent on a ~27K-token prompt) - negligible next to network latency - so it is always on. +The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency, so it is always on. ## Next Steps From 3784e9582fd43f9bb78edc4d19456dbc896b88f3 Mon Sep 17 00:00:00 2001 From: george larson Date: Mon, 24 Aug 2026 17:25:40 -0400 Subject: [PATCH 3/4] docs(sdk): note subscription-mode bucket behavior for prompt composition Mirrors the SDK docstring caveat (OpenHands/software-agent-sdk#4623): subscription mode folds the system prompt into the first user message, so those tokens count as history/latest on that transport. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 2203632c2..063177b3c 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -65,6 +65,7 @@ Composition counts are **client-side estimates**, computed with the model's toke - Tool schema counts follow litellm's `token_counter` serialization convention for tools, which can differ from the provider's wire-format tokenization. - For models litellm has no tokenizer mapping for, counts use litellm's fallback tokenizer and may deviate more from the provider's counts. - When token counting fails or is disabled (for example `litellm.disable_token_counter`), composition recording is skipped for that call; the call itself is unaffected. +- Buckets follow the wire, not the logical prompt: in subscription mode the system prompt is folded into the first user message before transport, so those tokens are counted in `history_tokens` or `latest_message_tokens` rather than `system_prompt_tokens` on that path. Treat the composition as a breakdown of *where* prompt tokens go, and `token_usages` as the record of *how many* tokens the provider billed. From d62f87ec426f3a935b679d4ef4a62043dbf0257a Mon Sep 17 00:00:00 2001 From: george larson Date: Tue, 25 Aug 2026 03:38:22 -0400 Subject: [PATCH 4/4] docs(sdk): make prompt composition opt-in via enable_prompt_composition Address rajshah4's review on software-agent-sdk#4623: composition recording is now opt-in (default off), so the page leads with enabling the flag and examples show it; the metrics field list notes the gate. Co-authored-by: openhands --- sdk/guides/llm-prompt-composition.mdx | 16 ++++++++++++---- sdk/guides/metrics.mdx | 2 +- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/sdk/guides/llm-prompt-composition.mdx b/sdk/guides/llm-prompt-composition.mdx index 063177b3c..81dbc717e 100644 --- a/sdk/guides/llm-prompt-composition.mdx +++ b/sdk/guides/llm-prompt-composition.mdx @@ -5,18 +5,26 @@ description: Break down prompt tokens per LLM call into system prompt, tool sche ## Overview -Every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: +When enabled, every LLM call records a per-call decomposition of its prompt tokens, so you can see where the input budget goes on each step of an agent run: - `system_prompt_tokens` - Tokens in system messages - `tool_tokens` - Tokens in the tool schemas included with the call - `history_tokens` - Tokens in conversation history (all non-system messages except the latest one) - `latest_message_tokens` - Tokens in the latest observation or user message -The decomposition is computed automatically on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). No configuration is required. On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. +Recording is **opt-in**: set `enable_prompt_composition=True` on the LLM. When off (the default), no tokenization pass runs and no records are appended. When on, the decomposition is computed on every call - both the Chat Completions and Responses API paths - and recorded into the LLM's [Metrics](/sdk/guides/metrics). On the Responses API path, the decomposition is computed on the finalized payload (instructions plus input items), so the record reflects what the provider received. + +```python icon="python" focus={4} +llm = LLM( + model="anthropic/claude-sonnet-4-5-20250929", + api_key=SecretStr(os.getenv("LLM_API_KEY")), + enable_prompt_composition=True, +) +``` ## Accessing the Composition -Each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition`: +With the flag enabled, each call appends one `PromptComposition` record to `llm.metrics.prompt_compositions`. The most recent record is available as `llm.metrics.latest_prompt_composition` (`None` when the flag is off): ```python icon="python" conversation.run() @@ -73,7 +81,7 @@ Composition counts are **client-side estimates**, computed with the model's toke ## Performance -The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency, so it is always on. +The estimator's cost scales linearly with prompt size: roughly 10-20 ms per call on a typical agent payload, measured at ~31 ms for a ~100K-token prompt and ~61 ms for ~190K tokens (19 tools, gpt-4o tokenizer) - negligible next to network latency. Because the feature is opt-in, this cost is only paid when `enable_prompt_composition=True`. ## Next Steps diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index e4035318d..ecd44ca3e 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -41,7 +41,7 @@ The `llm.metrics` object is an instance of the [Metrics class](https://github.co - `costs` - List of individual cost records per API call - `token_usages` - List of detailed token usage records per API call - `response_latencies` - List of response latency metrics per API call -- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message) +- `prompt_compositions` - List of per-call [prompt token composition](/sdk/guides/llm-prompt-composition) estimates (system prompt, tool schemas, history, latest message); populated only when `enable_prompt_composition=True` on the LLM For more details on the available metrics and methods, refer to the [source code](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-sdk/openhands/sdk/llm/utils/metrics.py).