From 6b5c51ec80980a3cac6b1b2b9ca5a35b91aeaa66 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 10:42:09 -0700 Subject: [PATCH 1/3] docs: focus codecs on provider normalization Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/codecs.mdx | 105 +++++++++++++--------- 1 file changed, 64 insertions(+), 41 deletions(-) diff --git a/docs/about-nemo-relay/concepts/codecs.mdx b/docs/about-nemo-relay/concepts/codecs.mdx index 0049d840e..6367bb14e 100644 --- a/docs/about-nemo-relay/concepts/codecs.mdx +++ b/docs/about-nemo-relay/concepts/codecs.mdx @@ -1,8 +1,10 @@ --- -title: "Codecs" -description: "" +title: "Provider Codecs" +description: "Understand provider request and response normalization, lossless patching, and semantic support boundaries." position: 7 --- +import { MermaidStyles } from "@/components/MermaidStyles"; + {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} @@ -11,48 +13,37 @@ This page explains how codecs fit into the shared NeMo Relay runtime contract. ## Overview A codec is a boundary translator. It converts one runtime-facing data shape into -another without changing who owns execution. +another without changing who owns execution. In the runtime model, the primary +role is [provider payload normalization](/integrate-into-frameworks/provider-codecs): +translating provider-native LLM requests and responses into stable annotations +that middleware and observability can use. NeMo Relay uses codecs when runtime behavior needs stable, JSON-compatible, or annotated data but the application or provider surface starts from a different -shape. +shape. A secondary [typed value codec](/integrate-into-frameworks/using-codecs) +path translates application objects at a public wrapper boundary. ## Key Features -Codecs let NeMo Relay preserve one execution model across different boundaries: +Provider codecs let NeMo Relay preserve one execution model across different +provider APIs: -- Application-owned typed values -- Framework-owned callback payloads - Provider-native LLM request and response shapes -- Exporter or subscriber consumers that need normalized data - -Without codecs, request-side middleware and observability would need to reason -about every framework or provider shape directly. - -## Two Main Codec Roles - -NeMo Relay documentation uses the word `codec` in two ways: typed value codecs -and provider codecs. They are related, but they differ in the ways described -below. - -### Typed Value Codecs +- Request middleware that needs normalized instructions, messages, tools, or + generation controls +- Events, subscribers, and exporters that need normalized response facts -Typed value codecs translate application-facing values to and from JSON-friendly -shapes at the public wrapper boundary. +Without provider codecs, request middleware and observability would need to +reason about every provider shape directly. -Typed value codecs are suitable for: - -- Application code wants native objects -- Framework callbacks expect typed values -- Runtime events and JSON-based middleware still need stable serialized payloads - -### Provider Codecs +## Provider Request and Response Codecs Provider codecs translate provider-native LLM payloads in the request and response halves of a provider call. First, request codecs decode provider requests into annotated request data for request intercepts and request-side middleware, then encode edits back into the provider shape when execution -continues. Later, after the provider returns, response codecs decode provider +continues. Later, after the provider returns, +[response codecs](/integrate-into-frameworks/provider-response-codecs) decode provider responses into annotated response data for LLM end events, subscribers, exporters, and diagnostics. @@ -63,6 +54,29 @@ Provider codecs are suitable for: - Response annotations such as usage, model names, or tool calls should be exposed in one stable shape for downstream consumers + + +### Request Path + +```mermaid +flowchart LR + native["Provider-native request"] --> decode["Request codec decodes"] + decode --> annotated["Annotated request"] + annotated --> middleware["Request middleware reads or edits"] + middleware --> encode["Request codec patches original"] + encode --> provider["Provider call"] +``` + +### Response Path + +```mermaid +flowchart LR + provider["Provider response"] --> decode["Response codec decodes"] + decode --> annotated["Annotated response"] + annotated --> event["LLM end event"] + event --> consumers["Subscribers and exporters"] +``` + Response decoding improves observability and downstream consistency. It does not automatically change the value returned to the application unless a separate typed value boundary also does so. @@ -97,6 +111,27 @@ A provider-native component can only be encoded by the provider surface named in its `provider` field. Provider mismatches and portable edits that the target API cannot represent fail before the provider callback. +### Preservation Is Not Semantic Support + +Provider APIs can add fields or change semantics before Relay's built-in codecs +are updated. The lossless patch contract preserves unknown fields and +provider-native components when they are unchanged, which keeps newer payloads +from being discarded. That pass-through behavior does not mean Relay +understands, validates, or can portably edit those fields. + +Semantic support is limited to the provider fields modeled by the current codec +implementation. New provider behavior requires a codec and test update before +middleware or exporters should rely on its normalized meaning. + +## Typed Value Codecs + +Typed value codecs are a separate wrapper concern. They translate +application-facing objects to and from JSON-compatible values when application +code or framework callbacks need native types but Relay events and middleware +need a stable serialized payload. Refer to +[Using Codecs](/integrate-into-frameworks/using-codecs) for implementation +guidance. + ## Normalized Data Consumption Normalized codec output is applicable to several runtime layers: @@ -206,15 +241,3 @@ Codecs do not decide: Those responsibilities belong to scopes, middleware, plugins, and exporter or subscriber registration. - -## Read Next - -- Use [Using Codecs](/integrate-into-frameworks/using-codecs) for typed value - codecs at framework-facing boundaries. -- Use [Provider Codecs](/integrate-into-frameworks/provider-codecs) for - provider-native request and response normalization. -- Use [Provider Response - Codecs](/integrate-into-frameworks/provider-response-codecs) when the main - need is response-side annotations for subscribers or exporters. -- Refer to the [Glossary](/resources/glossary) for the stable terminology used - across codecs, providers, and observability surfaces. From 8dd544eca7be895e1b9a486a238a544e896daa63 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 13:19:54 -0700 Subject: [PATCH 2/3] docs: clarify provider codec terminology Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/codecs.mdx | 96 +++++++++++------------ 1 file changed, 47 insertions(+), 49 deletions(-) diff --git a/docs/about-nemo-relay/concepts/codecs.mdx b/docs/about-nemo-relay/concepts/codecs.mdx index 6367bb14e..b7e0558cf 100644 --- a/docs/about-nemo-relay/concepts/codecs.mdx +++ b/docs/about-nemo-relay/concepts/codecs.mdx @@ -1,6 +1,6 @@ --- title: "Provider Codecs" -description: "Understand provider request and response normalization, lossless patching, and semantic support boundaries." +description: "Understand how provider codecs normalize requests and responses, preserve unknown fields, and expose supported fields." position: 7 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -12,47 +12,45 @@ This page explains how codecs fit into the shared NeMo Relay runtime contract. ## Overview -A codec is a boundary translator. It converts one runtime-facing data shape into -another without changing who owns execution. In the runtime model, the primary -role is [provider payload normalization](/integrate-into-frameworks/provider-codecs): -translating provider-native LLM requests and responses into stable annotations -that middleware and observability can use. +A provider codec translates provider-native LLM request and response payloads +into annotated Relay data. Middleware and observability can then work with +consistent fields without taking control of provider execution. Refer to +[Provider Payload Normalization](/integrate-into-frameworks/provider-codecs) for +implementation guidance. -NeMo Relay uses codecs when runtime behavior needs stable, JSON-compatible, or -annotated data but the application or provider surface starts from a different -shape. A secondary [typed value codec](/integrate-into-frameworks/using-codecs) -path translates application objects at a public wrapper boundary. +NeMo Relay also supports +[typed value codecs](/integrate-into-frameworks/using-codecs), which translate +application objects at a public wrapper API. ## Key Features -Provider codecs let NeMo Relay preserve one execution model across different -provider APIs: +Provider codecs handle: -- Provider-native LLM request and response shapes +- Provider-native LLM request and response payloads - Request middleware that needs normalized instructions, messages, tools, or generation controls - Events, subscribers, and exporters that need normalized response facts Without provider codecs, request middleware and observability would need to -reason about every provider shape directly. +parse every provider payload directly. ## Provider Request and Response Codecs Provider codecs translate provider-native LLM payloads in the request and response halves of a provider call. First, request codecs decode provider requests into annotated request data for request intercepts and request-side -middleware, then encode edits back into the provider shape when execution +middleware, then encode edits back into the provider payload when execution continues. Later, after the provider returns, [response codecs](/integrate-into-frameworks/provider-response-codecs) decode provider responses into annotated response data for LLM end events, subscribers, exporters, and diagnostics. -Provider codecs are suitable for: +Use provider codecs when: -- Provider payloads differ structurally -- Request intercepts need normalized request meaning +- Provider payloads differ structurally. +- Request intercepts need consistent request fields. - Response annotations such as usage, model names, or tool calls should be - exposed in one stable shape for downstream consumers + exposed through consistent fields for downstream consumers. @@ -79,7 +77,7 @@ flowchart LR Response decoding improves observability and downstream consistency. It does not automatically change the value returned to the application unless a separate -typed value boundary also does so. +typed value codec also does so. The built-in request codecs are lossless patch codecs. For an unchanged annotation, the following identity holds at the JSON-value level: @@ -100,14 +98,14 @@ changing a key overlays it. - `instructions` represents Anthropic `system` and OpenAI Responses `instructions`. - `messages`, content parts, function calls and results, tools, and tool choice - expose portable components when the shapes have shared semantics. -- `api_specific` is a tagged, mutable surface for modeled Anthropic Messages, + expose portable components when the provider formats have equivalent meaning. +- `api_specific` is a tagged, mutable field for modeled Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses controls. - Provider-only messages, content blocks, input items, tools, and tool choices use `{ provider, kind, value }`, where `value` is the exact native JSON. - Top-level `extra` is reserved for unknown future fields. -A provider-native component can only be encoded by the provider surface named +A provider-native component can only be encoded by the provider API named in its `provider` field. Provider mismatches and portable edits that the target API cannot represent fail before the provider callback. @@ -125,46 +123,46 @@ middleware or exporters should rely on its normalized meaning. ## Typed Value Codecs -Typed value codecs are a separate wrapper concern. They translate +Typed value codecs serve a different purpose. They translate application-facing objects to and from JSON-compatible values when application code or framework callbacks need native types but Relay events and middleware need a stable serialized payload. Refer to [Using Codecs](/integrate-into-frameworks/using-codecs) for implementation guidance. -## Normalized Data Consumption +## Where Normalized Data Is Used -Normalized codec output is applicable to several runtime layers: +Several parts of Relay use normalized codec output: -- Request intercepts or request-side middleware that need stable request meaning +- Request intercepts or request-side middleware that need consistent request fields - Lifecycle events that should expose consistent semantic payloads - Subscribers that inspect runtime activity in process - Exporters that write raw ATOF events or project them into ATIF or typed OpenTelemetry output Codecs do not replace scopes, middleware, subscribers, or plugins. They make -those layers easier to apply consistently across heterogeneous inputs. +those layers easier to apply consistently across provider-specific inputs. -## Extraction Strategy Boundaries +## Extraction Responsibilities -NeMo Relay keeps extraction responsibilities separated so refactors can reuse -normalization logic without changing runtime ownership or public binding APIs. +NeMo Relay separates extraction responsibilities so normalization logic can be +reused without changing public binding APIs. ### Provider Schema Extraction -Provider schema extraction is codec-owned. Built-in provider surfaces, such as -OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, each own the -logic that recognizes their request and response shapes and maps them into -`AnnotatedLlmRequest` or `AnnotatedLlmResponse`. +Provider codecs extract provider schema fields. The built-in codecs for OpenAI +Chat Completions, OpenAI Responses, and Anthropic Messages recognize their +request and response payloads and map them into `AnnotatedLlmRequest` or +`AnnotatedLlmResponse`. When a managed LLM event already has an annotation, subscribers and exporters consume that annotation. When an event has only raw provider JSON, best-effort -normalization may detect a built-in provider surface and decode it. This +normalization can detect a built-in provider codec and decode it. This fallback is fail-open: unrecognized, ambiguous, missing, sparse, or invalid payloads remain observable as raw lifecycle data. A recognized provider hint can -disambiguate an otherwise identical request shape, such as an Anthropic Messages +disambiguate an otherwise identical request payload, such as an Anthropic Messages request without a top-level `system` field, but no provider annotation is -invented without either a matching provider surface or a recognized hint. +invented without either a matching provider codec or a recognized hint. Provider extraction covers model names, instructions, messages, generation parameters, tool definitions, tool calls, finish reasons, usage, cost, @@ -180,24 +178,24 @@ usage, cost, provider-specific, and preserved `extra` fields. Cost parsing and estimation helpers are codec implementation details behind that interface, not a separate provider-response API. -### Provider Request Extraction +### Gateway Request Extraction -Provider request extraction is gateway-owned. It uses the selected gateway route, +The gateway extracts route-specific request facts. It uses the selected route, such as OpenAI Responses, OpenAI Chat Completions, OpenAI Models, Anthropic Messages, or Anthropic Count Tokens, to extract request facts that are not codec schema annotations. Route-specific request extractors resolve gateway session IDs, request-affinity keys, and fallback turn input for provider calls that arrive before the matching -agent prompt hook. This keeps correlation and ownership hints near gateway -alignment, while provider codecs stay focused on decoding request and response +agent prompt hook. This keeps correlation and routing hints in the gateway +routing logic, while provider codecs stay focused on request and response schemas. Provider request extraction can also pass a narrow provider hint into codec normalization. For example, the recognized `anthropic` and `anthropic.messages` hints let Anthropic Messages requests without a top-level `system` field decode -through the Anthropic provider surface instead of being treated as shape-only -OpenAI Chat payloads. +through the Anthropic codec instead of being inferred as OpenAI Chat payloads +from their fields alone. The `nemo-relay` gateway always enables matching request codecs for `/v1/messages`, `/v1/chat/completions`, and `/v1/responses`, for both buffered @@ -211,16 +209,16 @@ remains writable on routes without a request codec. Agent payload extraction is separate from provider codecs. Coding agents, harnesses, and framework hooks can expose session IDs, event names, subagent relationships, tool IDs, tool names, tool arguments, tool results, LLM hints, -and status fields through host-specific payload shapes. These facts help NeMo +and status fields through host-specific payload formats. These facts help NeMo Relay attach lifecycle events to the right scope, but they do not decode provider schemas or build request-affinity keys from provider requests. Agent extraction may be partial. Missing identifiers use compatibility -fallbacks at the adapter boundary, such as synthetic session IDs, synthetic tool +fallbacks in the adapter, such as synthetic session IDs, synthetic tool call IDs, an explicit `unknown_tool` name, or a generic subagent ID. Lossy, summary-only, or truncated payloads should keep their original payload and -metadata available for debugging instead of pretending to be reconstruction -grade provider data. +metadata available for debugging instead of presenting them as complete provider +data. ### Exporter Projection @@ -234,7 +232,7 @@ semantic attributes remain exporter-local. Codecs do not decide: -- Ownership boundaries +- Which scope owns the call - Middleware ordering - Whether execution is allowed to continue - Which exporter is active From 0b7f2d5c4924f0c4054894e291b01e7d3d035805 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 16:18:32 -0700 Subject: [PATCH 3/3] docs: clarify codec paths and extra fields Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/codecs.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/concepts/codecs.mdx b/docs/about-nemo-relay/concepts/codecs.mdx index b7e0558cf..0529f7fe0 100644 --- a/docs/about-nemo-relay/concepts/codecs.mdx +++ b/docs/about-nemo-relay/concepts/codecs.mdx @@ -56,6 +56,8 @@ Use provider codecs when: ### Request Path +The following diagram shows the request path. + ```mermaid flowchart LR native["Provider-native request"] --> decode["Request codec decodes"] @@ -67,6 +69,8 @@ flowchart LR ### Response Path +The following diagram shows the response path. + ```mermaid flowchart LR provider["Provider response"] --> decode["Response codec decodes"] @@ -103,7 +107,7 @@ changing a key overlays it. OpenAI Chat Completions, or OpenAI Responses controls. - Provider-only messages, content blocks, input items, tools, and tool choices use `{ provider, kind, value }`, where `value` is the exact native JSON. -- Top-level `extra` is reserved for unknown future fields. +- Top-level `extra` preserves unknown or unmodeled fields. A provider-native component can only be encoded by the provider API named in its `provider` field. Provider mismatches and portable edits that the target