From e367c534e1ef6081ebd87c2e583813b189fdffb1 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 10:32:06 -0700 Subject: [PATCH 1/3] docs: clarify Relay ecosystem boundaries Signed-off-by: Alex Fournier --- docs/about-nemo-relay/ecosystem.mdx | 104 ++++++++++++++++------------ 1 file changed, 58 insertions(+), 46 deletions(-) diff --git a/docs/about-nemo-relay/ecosystem.mdx b/docs/about-nemo-relay/ecosystem.mdx index afabe2329..c234b909f 100644 --- a/docs/about-nemo-relay/ecosystem.mdx +++ b/docs/about-nemo-relay/ecosystem.mdx @@ -22,20 +22,23 @@ Use this page to understand where NeMo Relay fits: - Across the Rust, Python, Node.js, Go, and C FFI surfaces in this repository -## How NeMo Relay Fits In The NVIDIA NeMo Ecosystem +## How NeMo Relay Fits in the NVIDIA NeMo Ecosystem The NVIDIA NeMo ecosystem spans model development, agent construction, guardrailing, inference, optimization, and runtime operations. NeMo Relay has a -narrower responsibility: it is the portable execution substrate that agent -systems can call when actual work crosses a scope, tool, or model boundary. +narrower responsibility: it is a shared execution runtime for scope, tool, and +model boundaries. -| Layer | Typical Responsibility | NeMo Relay Relationship | +| System | Role | How It Connects to NeMo Relay | |---|---|---| -| NeMo model, inference, and deployment components | Provide or serve the models an agent uses. | NeMo Relay records and controls LLM execution boundaries, but it does not train, host, or route model inference by itself. | -| NeMo Agent Toolkit and agent application frameworks | Build, run, profile, and optimize agent workflows across tools, data sources, and framework choices. | NeMo Relay can sit below these systems as the shared runtime contract for scopes, middleware, lifecycle events, subscribers, and plugins. | -| NeMo Guardrails and policy systems | Define safety, control, and compliance behavior for LLM applications. | NeMo Relay can host runtime guardrails and intercepts around managed tool and LLM calls, while higher-level guardrail systems can still own policy authoring and orchestration. | +| [NVIDIA NeMo Framework](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/index.html) | Build and customize generative AI models. | Applications can instrument calls to models produced with NeMo Framework. Relay does not train or customize the model. | +| [NVIDIA NIM for Large Language Models](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html) | Serve LLM inference through production endpoints. | Relay can observe or control the client-side LLM call boundary that sends requests to a NIM endpoint. Relay does not host the endpoint. | +| [NVIDIA Dynamo](https://docs.nvidia.com/dynamo) | Deploy and scale distributed inference services. | Applications can send Relay-managed model calls to a Dynamo-served endpoint. Relay does not schedule or operate the inference workers. | +| [NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/) and agent application frameworks | Build, run, profile, and optimize agent workflows across tools, data sources, and framework choices. | A framework integration can hand managed tool or LLM callbacks to Relay, or replay lifecycle hooks when the framework retains execution ownership. | +| [NeMo Guardrails](https://docs.nvidia.com/nemo-guardrails/index.html) and policy systems | Define safety, control, and compliance behavior for LLM applications. | Relay can run configured guardrails and intercepts around managed tool and LLM calls while the policy system owns policy authoring. | | Application harnesses and workflow code | Decide the agent pattern, planner, memory, retries, scheduling, and user-facing behavior. | NeMo Relay instruments the execution boundaries that the harness already owns. | -| Observability and evaluation backends | Store traces, trajectories, metrics, and analysis data. | NeMo Relay emits lifecycle events and exports them to in-process subscribers, Agent Trajectory Observability Format (ATOF), Agent Trajectory Interchange Format (ATIF), typed OpenTelemetry projections including OpenInference-compatible traces, or other backends. | +| Observability backends | Store and query traces, logs, and metrics. | Relay exporters can project lifecycle events into OpenTelemetry or OpenInference-compatible traces and send them through an OTLP pipeline. | +| Trajectory and evaluation systems | Replay or evaluate completed agent runs. | Relay can project lifecycle events into Agent Trajectory Interchange Format (ATIF) artifacts for offline analysis, replay, or evaluation. | In practical terms, NeMo Relay answers a different question than higher-level agent products. A framework asks, "What should the agent do next?" NeMo Relay @@ -51,8 +54,8 @@ different roles and can be used alongside Relay. | Tooling | Primary Role | Relationship to NeMo Relay | |---|---|---| -| OpenTelemetry GenAI conventions | Define a common representation for AI telemetry. | Relay owns runtime capture and control, then can export its lifecycle events as generic OTLP spans for an existing OpenTelemetry tracing pipeline. | -| Langfuse, LangSmith, and Arize Phoenix | Store, explore, and evaluate traces and agent runs. | These products are outside Relay's execution control loop. Use Relay to establish consistent execution boundaries and event data, then send data through the configured subscriber or export path that fits your backend. | +| [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) | Define common attributes and span conventions for AI telemetry. | Relay owns runtime capture and control, then can export lifecycle events through its typed OpenTelemetry projections. | +| Langfuse, LangSmith, Arize Phoenix, and other observability products | Store and explore traces or agent runs. | These products stay outside Relay's execution control loop and consume data from a configured export path. | Relay is not a replacement for a telemetry standard or an observability backend. Its role is to make the real execution path observable and controllable before @@ -62,71 +65,80 @@ the resulting lifecycle data is stored, visualized, or evaluated elsewhere. ```mermaid flowchart TB - User[User / Application] - Framework[Agent Framework or Harness] - Toolkit[NeMo Agent Toolkit / Framework Integrations] + App[Application] + Framework[Agent Toolkit / Framework / Harness] Relay[NeMo Relay Runtime] - Provider[Model, Tool, or Provider SDK] - Obs[Subscribers and Observability Backends] + Provider[Model or Tool Provider] Policy[Guardrails, Intercepts, and Plugins] - - User --> Framework - Framework --> Toolkit - Toolkit --> Relay - Framework -. direct instrumentation .-> Relay + Events[Canonical Lifecycle Events] + + subgraph ObservabilityFlow[Observability Flow] + OTel[OpenTelemetry or OpenInference Exporter] + Collector[OTLP Collector] + Backend[Observability Backend] + OTel --> Collector --> Backend + end + + subgraph EvaluationFlow[Trajectory and Evaluation Flow] + ATIF[ATIF Exporter] + Evaluation[Replay or Evaluation System] + ATIF --> Evaluation + end + + App --> Framework --> Relay + App -. direct instrumentation .-> Relay Relay --> Provider Policy --> Relay - Relay --> Obs + Relay --> Events + Events --> OTel + Events --> ATIF class Framework yellow-lightest; - class Toolkit blue-lightest; class Relay green-light; class Provider purple-lightest; - class Obs grey-light; class Policy green-lightest; + class Events blue-lightest; + class ObservabilityFlow,EvaluationFlow grey-lightest; ``` -The dotted path matters. An application or custom harness can call NeMo Relay -directly without adopting a higher-level framework. A framework integration can -also call NeMo Relay on behalf of application code when the framework owns the -tool or provider boundary. +The dotted path shows that an application can call Relay directly without +adopting a higher-level framework. In the observability flow, Relay projects +events into traces and sends them through an OTLP collector. In the trajectory +flow, Relay writes ATIF artifacts for replay or evaluation. -## How NeMo Relay Fits Agent Frameworks And Harnesses +## How NeMo Relay Fits Agent Frameworks and Harnesses The agent framework and harness landscape is intentionally mixed. A team might use NeMo Agent Toolkit, LangChain, LangGraph, an internal orchestration layer, a provider SDK, or direct application code. NeMo Relay is designed to meet those systems at stable execution boundaries instead of requiring one framework shape. -| Integration Point | Use NeMo Relay For | Keep In The Framework Or Harness | -|---|---|---| -| Request, run, workflow, or agent lifecycle hooks | Create scopes, emit scope start and end events, and isolate concurrent work. | Scheduling, routing, retry policy, planner choice, memory, and user session state. | -| Tool invocation callbacks | Run managed tool execution, apply tool middleware, emit tool lifecycle events, and preserve parent scope context. | Tool discovery, tool schema presentation, framework-specific callback signatures, and application-visible result handling. | -| LLM or provider adapter calls | Run managed LLM execution, attach model metadata, apply LLM middleware, handle stream lifecycle events, and emit normalized observability payloads. | Provider clients, authentication, transport, provider-native request objects, and provider-specific response types. | -| Framework internals that cannot hand over a callback | Use explicit lifecycle APIs, request-intercept helpers, guardrail helpers, or mark events. | The actual invocation path when the framework must retain control. | -| Cross-cutting behavior | Package middleware, subscribers, adaptive behavior, and reusable policy as plugins. | Framework configuration, agent definitions, deployment topology, and business logic. | +The framework or harness keeps ownership of: + +- Agent orchestration, planning, memory, retries, and scheduling. +- Tool discovery, schemas, and application-visible results. +- Provider clients, authentication, transport, and provider-native objects. +- Public callback signatures and framework-specific behavior. + +Relay receives: + +- Scope boundaries and parentage for work that should be observed. +- A real tool or LLM callback when Relay can own managed execution. +- Start and end lifecycle notifications when the framework must retain control. +- JSON-compatible observability payloads and metadata for events and middleware. Prefer a managed execution wrapper when a framework exposes a stable callback that NeMo Relay can own. Use explicit lifecycle calls or standalone helpers when the framework owns the callback internally but exposes reliable start, finish, or request transformation hooks. -This lets NeMo Relay provide consistent runtime semantics without forcing a -framework migration: - -- Applications keep their existing agent orchestration model -- Framework adapters preserve public behavior and callback signatures -- Non-serializable provider objects stay in framework-owned storage -- NeMo Relay receives JSON-compatible payloads for middleware and events -- Subscribers see a consistent scope, tool, and LLM event stream across integrations +This boundary lets subscribers see a consistent scope, tool, and LLM event +stream without changing the framework's public behavior. ## Related Topics Use these links to continue into adjacent concepts and workflows. -- [NVIDIA NeMo documentation](https://docs.nvidia.com/nemo/index.html) -- [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/) -- [NVIDIA NeMo Guardrails documentation](https://docs.nvidia.com/nemo-guardrails/index.html) - [Integrate into Frameworks](/integrate-into-frameworks/about) - [Adding Framework Scopes](/integrate-into-frameworks/adding-scopes) - [Wrapping Tool Calls](/integrate-into-frameworks/wrap-tool-calls) From db62dca25021673a13af93fcdd31368ef0602871 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 13:19:54 -0700 Subject: [PATCH 2/3] docs: clarify ecosystem roles Signed-off-by: Alex Fournier --- docs/about-nemo-relay/ecosystem.mdx | 58 ++++++++++++++--------------- 1 file changed, 27 insertions(+), 31 deletions(-) diff --git a/docs/about-nemo-relay/ecosystem.mdx b/docs/about-nemo-relay/ecosystem.mdx index c234b909f..e0e4b064b 100644 --- a/docs/about-nemo-relay/ecosystem.mdx +++ b/docs/about-nemo-relay/ecosystem.mdx @@ -8,7 +8,6 @@ import { MermaidStyles } from "@/components/MermaidStyles"; {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} - NeMo Relay is the agent execution runtime layer in the NVIDIA NeMo ecosystem. It does not replace an agent framework, model provider, guardrail authoring system, or deployment platform. Instead, it gives those systems one shared way to model @@ -19,24 +18,24 @@ Use this page to understand where NeMo Relay fits: - Inside the NVIDIA NeMo software stack - Inside agent frameworks, harnesses, and provider adapters -- Across the Rust, Python, Node.js, Go, and C FFI surfaces in this +- Across the Rust, Python, Node.js, Go, and C FFI bindings in this repository ## How NeMo Relay Fits in the NVIDIA NeMo Ecosystem The NVIDIA NeMo ecosystem spans model development, agent construction, guardrailing, inference, optimization, and runtime operations. NeMo Relay has a -narrower responsibility: it is a shared execution runtime for scope, tool, and -model boundaries. +narrower responsibility: it tracks and controls work across scopes, tools, and +models. | System | Role | How It Connects to NeMo Relay | |---|---|---| | [NVIDIA NeMo Framework](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/index.html) | Build and customize generative AI models. | Applications can instrument calls to models produced with NeMo Framework. Relay does not train or customize the model. | -| [NVIDIA NIM for Large Language Models](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html) | Serve LLM inference through production endpoints. | Relay can observe or control the client-side LLM call boundary that sends requests to a NIM endpoint. Relay does not host the endpoint. | +| [NVIDIA NIM for Large Language Models](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html) | Serve LLM inference through production endpoints. | Relay can observe or control client requests to a NIM endpoint. Relay does not host the endpoint. | | [NVIDIA Dynamo](https://docs.nvidia.com/dynamo) | Deploy and scale distributed inference services. | Applications can send Relay-managed model calls to a Dynamo-served endpoint. Relay does not schedule or operate the inference workers. | -| [NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/) and agent application frameworks | Build, run, profile, and optimize agent workflows across tools, data sources, and framework choices. | A framework integration can hand managed tool or LLM callbacks to Relay, or replay lifecycle hooks when the framework retains execution ownership. | +| [NeMo Agent Toolkit](https://docs.nvidia.com/nemo/agent-toolkit/latest/) and agent application frameworks | Build, run, profile, and optimize agent workflows across tools, data sources, and framework choices. | A framework integration can give Relay tool or LLM callbacks to manage, or emit lifecycle events when the framework invokes the callbacks itself. | | [NeMo Guardrails](https://docs.nvidia.com/nemo-guardrails/index.html) and policy systems | Define safety, control, and compliance behavior for LLM applications. | Relay can run configured guardrails and intercepts around managed tool and LLM calls while the policy system owns policy authoring. | -| Application harnesses and workflow code | Decide the agent pattern, planner, memory, retries, scheduling, and user-facing behavior. | NeMo Relay instruments the execution boundaries that the harness already owns. | +| Application harnesses and workflow code | Decide the agent pattern, planner, memory, retries, scheduling, and user-facing behavior. | NeMo Relay instruments the tool and model calls that the harness already invokes. | | Observability backends | Store and query traces, logs, and metrics. | Relay exporters can project lifecycle events into OpenTelemetry or OpenInference-compatible traces and send them through an OTLP pipeline. | | Trajectory and evaluation systems | Replay or evaluate completed agent runs. | Relay can project lifecycle events into Agent Trajectory Interchange Format (ATIF) artifacts for offline analysis, replay, or evaluation. | @@ -47,15 +46,15 @@ what events are emitted, and which subscribers can consume the result?" ## How NeMo Relay Relates to Other Tooling -NeMo Relay is an execution-time runtime contract. It captures work at the -scope, tool, and LLM boundaries, and can apply middleware before or around that -work. Telemetry conventions and observability or evaluation products serve +NeMo Relay runs alongside application code while the agent is executing. It +captures scope, tool, and LLM activity and can apply middleware before or around +that work. Telemetry conventions and observability or evaluation products serve different roles and can be used alongside Relay. | Tooling | Primary Role | Relationship to NeMo Relay | |---|---|---| -| [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) | Define common attributes and span conventions for AI telemetry. | Relay owns runtime capture and control, then can export lifecycle events through its typed OpenTelemetry projections. | -| Langfuse, LangSmith, Arize Phoenix, and other observability products | Store and explore traces or agent runs. | These products stay outside Relay's execution control loop and consume data from a configured export path. | +| [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) | Define common attributes and span conventions for AI telemetry. | Relay captures and controls runtime activity, then can export lifecycle events through its typed OpenTelemetry projections. | +| Langfuse, LangSmith, Arize Phoenix, and other observability products | Store and explore traces or agent runs. | These products do not run the agent's tool or model calls. They consume data from a configured export path. | Relay is not a replacement for a telemetry standard or an observability backend. Its role is to make the real execution path observable and controllable before @@ -111,36 +110,33 @@ flow, Relay writes ATIF artifacts for replay or evaluation. The agent framework and harness landscape is intentionally mixed. A team might use NeMo Agent Toolkit, LangChain, LangGraph, an internal orchestration layer, a provider SDK, or direct application code. NeMo Relay is designed to meet those -systems at stable execution boundaries instead of requiring one framework shape. +systems at tool, model, and lifecycle hooks, so they do not need to use one +common API. -The framework or harness keeps ownership of: +The framework or harness continues to manage: - Agent orchestration, planning, memory, retries, and scheduling. - Tool discovery, schemas, and application-visible results. - Provider clients, authentication, transport, and provider-native objects. - Public callback signatures and framework-specific behavior. -Relay receives: +Relay can receive: -- Scope boundaries and parentage for work that should be observed. -- A real tool or LLM callback when Relay can own managed execution. -- Start and end lifecycle notifications when the framework must retain control. +- The scope and parent for work that should be observed. +- A tool or LLM function when Relay can invoke it through managed execution. +- Start and end lifecycle notifications when the framework invokes the function. - JSON-compatible observability payloads and metadata for events and middleware. -Prefer a managed execution wrapper when a framework exposes a stable callback -that NeMo Relay can own. Use explicit lifecycle calls or standalone helpers when -the framework owns the callback internally but exposes reliable start, finish, or +Prefer a managed execution wrapper when a framework lets NeMo Relay invoke the +tool or LLM function. Use explicit lifecycle calls or standalone helpers when +the framework invokes the function but exposes reliable start, finish, or request transformation hooks. -This boundary lets subscribers see a consistent scope, tool, and LLM event +This arrangement lets subscribers see a consistent scope, tool, and LLM event stream without changing the framework's public behavior. -## Related Topics - -Use these links to continue into adjacent concepts and workflows. - -- [Integrate into Frameworks](/integrate-into-frameworks/about) -- [Adding Framework Scopes](/integrate-into-frameworks/adding-scopes) -- [Wrapping Tool Calls](/integrate-into-frameworks/wrap-tool-calls) -- [Wrapping LLM Calls](/integrate-into-frameworks/wrap-llm-calls) -- [Language Binding Plugins](/build-plugins/language-binding/about) +Refer to [Integrate into Frameworks](/integrate-into-frameworks/about) to choose +an integration method. Use [Adding Framework Scopes](/integrate-into-frameworks/adding-scopes) +for lifecycle hooks, or [Wrapping Tool Calls](/integrate-into-frameworks/wrap-tool-calls) +and [Wrapping LLM Calls](/integrate-into-frameworks/wrap-llm-calls) when Relay +can invoke the function. From e0674715790da2172a99b8e8774f75ee10b37913 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 16:18:26 -0700 Subject: [PATCH 3/3] docs: introduce the ecosystem diagram Signed-off-by: Alex Fournier --- docs/about-nemo-relay/ecosystem.mdx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/about-nemo-relay/ecosystem.mdx b/docs/about-nemo-relay/ecosystem.mdx index e0e4b064b..8dcc1bc04 100644 --- a/docs/about-nemo-relay/ecosystem.mdx +++ b/docs/about-nemo-relay/ecosystem.mdx @@ -62,6 +62,9 @@ the resulting lifecycle data is stored, visualized, or evaluated elsewhere. +The following diagram shows how Relay connects application execution to +observability and evaluation systems. + ```mermaid flowchart TB App[Application]