Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 76 additions & 44 deletions docs/about-nemo-relay/agent-runtime-primer.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import { MermaidStyles } from "@/components/MermaidStyles";
{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}


NeMo Relay is a portable runtime layer for agent systems that already have an
application, framework, or model provider. Use this primer when you need to
understand what NeMo Relay adds before choosing an installation, quick-start, or
Expand All @@ -18,7 +17,7 @@ Agent applications usually cross several boundaries in one request: an entry
point starts work, the agent calls a model, the model asks for tools, tools call
services, and tracing or policy systems need to understand the result. Without a
shared runtime layer, each boundary tends to grow its own wrappers, callback
shape, trace vocabulary, and cleanup rules.
format, trace vocabulary, and cleanup rules.

NeMo Relay gives those boundaries one execution model.

Expand All @@ -27,56 +26,89 @@ NeMo Relay gives those boundaries one execution model.
NeMo Relay does not decide what your agent should do. It describes and manages
what happens when your agent crosses runtime boundaries.

The shared runtime model has five parts, with managed calls and codecs acting
as paths through that model rather than separate model layers:
The shared runtime model has five parts:

- **Scopes** describe where work belongs. They preserve parent-child
- **[Scopes](/about-nemo-relay/concepts/scopes)** describe where work belongs.
They preserve parent-child
relationships across requests, agent runs, tools, LLM calls, background work,
and nested functions.
- **Middleware** runs around managed execution. Intercepts can transform or wrap
real calls. Guardrails can block execution or sanitize emitted observability
payloads.
- **Plugins** package reusable runtime behavior so teams can install middleware,
subscribers, exporters, or adaptive behavior from configuration instead of
repeating setup code in every application.
- **Events** record what happened. NeMo Relay emits Agent Trajectory
Observability Format (ATOF) lifecycle records that subscribers and exporters
can consume.
- **Subscribers and exporters** consume events in process, write raw ATOF
events, or project events into ATIF, typed OpenTelemetry output such as the
OpenInference projection, or other downstream formats.

Managed tool and LLM calls are the main application-owned API path through that
model: they attach work to the active scope, run middleware in a consistent
order, and emit lifecycle events. The application result is preserved unless
registered intercepts or guardrails intentionally change the execution path.

Codecs translate typed application values or provider-native payloads into
stable runtime shapes when request-side middleware, events, or exporters need
normalized data. They are boundary translators, not a separate execution model.

The simplest mental model is below. Codecs appear only when a boundary needs
payload normalization, so they are shown as an optional translator rather than a
required step for every call.
- **[Middleware](/about-nemo-relay/concepts/middleware)** runs around managed
execution. Intercepts can transform or wrap real calls. Guardrails can block
execution or sanitize emitted observability payloads.
- **[Plugins](/about-nemo-relay/concepts/plugins)** package reusable runtime
behavior so teams can install middleware, subscribers, exporters, or adaptive
behavior from configuration instead of repeating setup code in every
application.
- **[Events](/about-nemo-relay/concepts/events)** record what happened. NeMo
Relay emits Agent Trajectory Observability Format (ATOF) lifecycle records
that subscribers and exporters can consume.
- **[Subscribers and exporters](/about-nemo-relay/concepts/subscribers)**
consume events in process, write raw ATOF events, or project events into ATIF,
typed OpenTelemetry output such as the OpenInference projection, or other
downstream formats.

Managed tool and LLM calls are the main APIs for application-owned execution.
They attach work to the active scope, run middleware in a consistent order, and
emit lifecycle events. The application result is preserved unless registered
intercepts or guardrails intentionally change execution.

Choose a built-in plugin component when Relay already provides the behavior and
you want to enable it from configuration. Choose a discoverable plugin when a
separately distributed plugin package should install internal or third-party
behavior without changing the Relay host.

## One Agent Run, Two Views

Consider an agent that reads a file and then asks an LLM to summarize it:

1. Relay opens an Agent scope under the root scope.
2. The file read runs in a Tool scope under the Agent scope.
3. After the Tool scope ends, the model request runs in an LLM scope under the
same Agent scope.
4. Relay ends the Agent scope when the run completes.

Scopes are active as a stack. During the file read, the stack is
`root -> agent -> tool`. During the model request, it is
`root -> agent -> llm`. Ending the current Tool or LLM scope pops it from the
stack and restores the Agent scope as the active owner.

Events are recorded separately from the active stack. Relay emits parent-linked
start and end events for the Agent, Tool, and LLM scopes. Those records remain
in an event tree after the active scope has closed. A mark records a
point-in-time fact under the active scope; it does not add another entry to the
scope stack.

<MermaidStyles />

The following diagram contrasts active scope-stack snapshots with the
parent-linked event tree.

```mermaid
Comment thread
afourniernv marked this conversation as resolved.
flowchart LR
boundary["App or framework boundary"]
scope["NeMo Relay scope"]
managedCall["Managed tool or LLM call"]
codec["Codec boundary when needed"]
middleware["Middleware"]
event["Lifecycle event"]
sink["Subscriber or exporter"]

boundary --> scope --> managedCall --> middleware --> event --> sink
managedCall -. normalize payloads .-> codec
codec -. normalized data .-> middleware
codec -. annotations .-> event
subgraph Active[Active Scope Stack Snapshots]
toolStack["root -> agent -> tool"]
llmStack["root -> agent -> llm"]
toolStack -->|tool ends; LLM starts| llmStack
end

subgraph Recorded[Parent-Linked Event Tree]
agent["Agent scope events"]
tool["Tool scope events"]
llm["LLM scope events"]
mark["Mark event"]
agent --> tool
agent --> llm
tool --> mark
end

toolStack -. emits .-> tool
llmStack -. emits .-> llm
```

Codecs are optional translators. They convert typed application values or
provider-native payloads into consistent data that middleware, events, or
exporters can use. Codecs are not required for every call.

## What NeMo Relay Does Not Replace

NeMo Relay sits below the choices your application already makes.
Expand All @@ -89,7 +121,7 @@ It does not replace:
- Your production observability backend
- NeMo Agent Toolkit

Instead, it gives those systems a shared runtime contract for call boundaries,
policy hooks, event emission, and export.
Instead, it gives those systems shared handling for call lifecycles, policy
hooks, event emission, and export.

For setup routing, start with [Getting Started](/getting-started/about).