Skip to content

docs: update runtime architecture ownership #697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
afourniernv wants to merge 6 commits into
base: main
Choose a base branch
from
+187 −56
Open

docs: update runtime architecture ownership #697

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9e65526
docs: update runtime architecture ownership
afourniernv Aug 4, 2026
4c47523
docs: clarify runtime architecture
afourniernv Aug 4, 2026
178c80f
docs: link runtime model concepts
afourniernv Aug 4, 2026
749e913
docs: describe subscriber destinations
afourniernv Aug 4, 2026
326cb32
docs: add runtime architecture diagrams
afourniernv Aug 4, 2026
e7aa708
docs: clarify runtime teardown
afourniernv Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 187 additions & 56 deletions docs/about-nemo-relay/architecture.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,24 @@ import { MermaidStyles } from "@/components/MermaidStyles";
{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}

This page explains how NeMo Relay connects scopes, middleware, plugins, events,
subscribers, and exporters.
This page explains how NeMo Relay connects [scopes](/about-nemo-relay/concepts/scopes),
[middleware](/about-nemo-relay/concepts/middleware),
[plugins](/about-nemo-relay/concepts/plugins),
[events](/about-nemo-relay/concepts/events), and
[subscribers](/about-nemo-relay/concepts/subscribers).

## Architecture Diagram

This diagram connects the runtime pieces to the layers they inhabit.
This diagram shows how runtime hosts and integrations reach the shared Rust
runtime. Plugin components install reusable behavior, and subscribers deliver
event snapshots to in-process consumers or external backends.

<MermaidStyles />

```mermaid
flowchart TB
subgraph AppLayer[Framework Integrations and Application Code]
subgraph AppLayer[Runtime Hosts and Integrations]
CLI[NeMo Relay CLI]
App[Application Code]
Framework[Framework Integration]
end
Expand All @@ -28,9 +34,10 @@ flowchart TB
Bindings[Language Bindings]
end

subgraph PluginLayer[Plugin and Adaptive Layer]
subgraph PluginLayer[Plugin System and Components]
PluginSystem[Plugin System]
Adaptive[Adaptive Component]
Components[Adaptive / Observability / Guardrail Components]
PluginSystem -->|activates| Components
end

subgraph CoreLayer[Core Runtime]
Expand All @@ -54,10 +61,11 @@ flowchart TB
App -. direct use .-> Bindings
App -->|registers and configures| PluginSystem
Framework -->|calls| Bindings
CLI -->|hosts| Core
CLI -->|loads configuration| PluginSystem
Bindings --> Core
Adaptive -->|activates via| PluginSystem
PluginSystem -->|installs| Registry
PluginSystem -->|installs| Subs
Components -->|register middleware| Registry
Components -->|register subscribers| Subs
Core -->|updates| Scope
Core -->|resolves| Registry
Core -->|emits| Events
Expand All @@ -71,41 +79,55 @@ flowchart TB
class CoreLayer grey-hint;
class ObsLayer grey-hint;
class RuntimeState grey-lightest;
class CLI purple-lightest;
class App purple-lightest;
class Framework yellow-lightest;
class Bindings green-lightest;
class PluginSystem green-light;
class Adaptive blue-lightest;
class Components blue-lightest;
class Core green-light;
class Scope green-light;
class Registry green-light;
class Events green-light;
class Dispatcher green-light;
class Subs green-light;
class Backends grey-light;
```

Adaptive appears here as a built-in plugin component rather than a separate runtime model because it activates through the same plugin lifecycle.
Adaptive behavior is one plugin component installed through the same lifecycle
as observability or PII redaction. It is not a separate runtime layer.

## Runtime Model

NeMo Relay combines a small number of runtime pieces into one shared execution model:

- The **scope stack** answers where work belongs
- The **middleware registries** answer what should happen around that work
- The **plugin system** installs reusable runtime behavior from configuration
- The **event stream** records what happened
- The **async subscriber dispatcher** delivers event snapshots after emission
- **subscribers** consume those events

Every emitted scope, tool, LLM, or mark event attaches to the active scope stack. Every managed tool or LLM call resolves the currently visible middleware before it executes.
NeMo Relay combines a small number of runtime pieces into one shared execution
model:

- The [**scope stack**](/about-nemo-relay/concepts/scopes) answers where work
currently belongs.
- The [**middleware registries**](/about-nemo-relay/concepts/middleware) answer
what should happen around that work.
- The [**plugin system**](/about-nemo-relay/concepts/plugins) installs reusable
runtime behavior from configuration.
- The [**event records**](/about-nemo-relay/concepts/events) preserve what
happened and how work was related.
- The [**async subscriber dispatcher**](/about-nemo-relay/concepts/subscribers#waiting-for-delivery)
delivers event snapshots after emission.
- [**Subscribers and exporters**](/about-nemo-relay/concepts/subscribers#common-subscriber-roles)
consume those snapshots.

Every managed tool or LLM call resolves the middleware visible from the active
scope before it executes. When the runtime emits an event, it records the active
scope UUID as parentage. The scope stack changes as work opens and closes; the
parent-linked event records remain available to subscribers.

## Main Runtime Pieces

These components are the primary building blocks that make up the runtime model.

### Scope Stack

The active scope stack defines the ownership tree for runtime work. It establishes:
The active scope stack defines the current context for runtime work.
It establishes:

- Parent-child relationships between events
- Scope-local visibility for middleware and subscribers
Expand All @@ -114,21 +136,53 @@ The active scope stack defines the ownership tree for runtime work. It establish

### Middleware Registries

The middleware registries hold the active intercepts and guardrails for tool and LLM execution. Managed helpers read those registries before invoking the real callback.
The middleware registries hold the active intercepts and guardrails for tool and
LLM execution. Request intercepts can rewrite real requests, conditional
guardrails can reject execution, and sanitize guardrails can change emitted
observability payloads. Managed helpers read those registries before invoking
the real callback.

Async middleware callbacks are awaited as part of managed execution. The
managed call does not advance past that middleware callback until the callback
returns.

### Plugin System

The plugin system installs reusable runtime components from configuration. A plugin can register middleware, subscribers, or related behavior without requiring each application call site to do the work manually.
The plugin system installs reusable runtime components from configuration. A
plugin can register middleware, subscribers, or related behavior without
requiring each application call site to repeat the setup.

### Event Emission

The runtime emits structured events for scopes, tools, LLMs, and named marks. Those events are the canonical record of runtime behavior. Native Rust, Python, Node.js, and FFI event-producing APIs enqueue subscriber work and return without waiting for subscriber callbacks or exporter work.
The runtime emits structured events for scopes, tools, LLMs, and named marks.
Those parent-linked records are the canonical history of runtime behavior.
Native Rust, Python, Node.js, and FFI event-producing APIs enqueue subscriber
work and return without waiting for subscriber callbacks or exporter work.

### Subscribers and Exporters

Subscribers consume the event stream through the background dispatcher. Some subscribers stay in-process. Others export that stream into files or tracing systems. Use the binding flush API when a test or shutdown path must wait for already-queued subscriber work.

## Two Axes of Runtime State
Subscribers consume event snapshots through the background dispatcher. Some
stay in process. Exporter subscribers can write
[ATOF JSONL](/configure-plugins/observability/atof), project events into
[ATIF trajectories](/configure-plugins/observability/atif), or emit
[OpenTelemetry traces](/configure-plugins/observability/opentelemetry).
Typical destinations include in-process application logic, local files and
artifact pipelines, OTLP-compatible observability backends, and evaluation or
visualization tools.

This delivery happens after event submission and is separate from awaited async
middleware. Use the binding flush API when a test or shutdown path must wait for
already queued subscriber work. For manually registered exporters, follow the
exporter's documented teardown order before process exit. The
[ATOF](/configure-plugins/observability/atof),
[OpenTelemetry](/configure-plugins/observability/opentelemetry), and
[OpenInference](/configure-plugins/observability/openinference) exporters flush,
deregister, and then shut down. The
[ATIF](/configure-plugins/observability/atif) exporter drains subscriber work,
exports, deregisters, and then clears its state. Clearing plugin configuration
Comment on lines +175 to +182

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Correct the ATOF teardown sequence.

Line 179 states flush, deregister, then shutdown. go/nemo_relay/atof_test.go uses Deregister, ForceFlush, then Shutdown. Document that order for ATOF. Verify the OpenTelemetry, OpenInference, and ATIF sequences independently before grouping them under one lifecycle statement.

Proposed correction 
-The [ATOF](/configure-plugins/observability/atof),
-[OpenTelemetry](/configure-plugins/observability/opentelemetry), and
-[OpenInference](/configure-plugins/observability/openinference) exporters flush,
-deregister, and then shut down.
+For ATOF, deregister the exporter, force-flush queued work, and then shut down.

As per coding guidelines, “Examples and documentation must use each exporter's documented flush/deregister order before shutdown.” As per path instructions, “Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/about-nemo-relay/architecture.mdx` around lines 175 - 182, Correct the
exporter teardown documentation around the ATOF link to state that ATOF
deregisters, force-flushes, and then shuts down, matching the current API
behavior. Independently verify the OpenTelemetry, OpenInference, and ATIF
teardown sequences before grouping them in a shared lifecycle statement;
preserve each exporter’s documented order.

Sources: Coding guidelines, Path instructions

owns teardown for plugin-installed exporters.

## Where Runtime State Lives

Runtime state is easiest to understand by separating ownership from process-wide
registration.
Expand All @@ -142,6 +196,29 @@ The scope stack defines:
- When scope-local registrations are cleaned up
- Whether concurrent requests stay isolated

Only scopes are pushed onto the stack. Managed LLM and tool calls emit lifecycle
records that use the current top scope as `parent_uuid`; they do not become
stack entries. This example shows an agent scope with a nested function scope
named `turn-a`.

```mermaid
flowchart BT
Root["BOTTOM: Implicit root<br/>uuid = root-a"]
Agent["Agent scope<br/>uuid = agent-a<br/>parent_uuid = root-a"]
Turn["TOP: Function scope (turn)<br/>uuid = turn-a<br/>parent_uuid = agent-a"]

Root -->|"push agent-a"| Agent
Agent -->|"push turn-a"| Turn

class Root grey-light;
class Agent,Turn green-light;
```

With `turn-a` at the top, managed LLM and tool records receive
`parent_uuid = turn-a`, so they are siblings in the event tree. Popping
`turn-a` returns `agent-a` to the top and removes registrations owned by the
turn. A concurrent request uses a separate stack.

### Middleware Ownership

Middleware exists at two levels:
Expand All @@ -151,6 +228,43 @@ Middleware exists at two levels:

That split lets long-lived defaults coexist with request-specific or task-specific behavior.

The following diagram shows how global and scope-local middleware are resolved
for two concurrent requests.

```mermaid
Comment thread
afourniernv marked this conversation as resolved.
flowchart TB
Global["Global middleware<br/>process-wide"]

subgraph RequestA["Request A scope stack"]
LocalA["Scope-local middleware<br/>owned by agent-a"]
ResolveA["Merge visible entries<br/>and order by priority"]
CallA["Managed call A"]
CloseA["Close agent-a"]
LocalA --> ResolveA --> CallA
CloseA -.->|"removes"| LocalA
end

subgraph RequestB["Request B scope stack"]
LocalB["Scope-local middleware<br/>owned by agent-b"]
ResolveB["Merge visible entries<br/>and order by priority"]
CallB["Managed call B"]
LocalB --> ResolveB --> CallB
end

Global --> ResolveA
Global --> ResolveB

class Global blue-lightest;
class RequestA,RequestB grey-lightest;
class LocalA,LocalB green-lightest;
class ResolveA,ResolveB green-light;
class CallA,CallB yellow-lightest;
class CloseA grey-light;
```

Global entries are visible to both calls. Each scope-local entry is visible
only through its owning stack and is removed when that scope closes.

## Managed Execution Pipeline

Managed tool and LLM execution follows the same high-level order:
Expand All @@ -162,36 +276,53 @@ Managed tool and LLM execution follows the same high-level order:
5. The user callback runs.
6. Sanitize-response guardrails can rewrite the emitted end-event payload.

The following sequence shows where each middleware family runs during a managed
call.

```mermaid
sequenceDiagram
autonumber
actor Caller as Application / Framework
participant Runtime as NeMo Relay Runtime
participant Conditional as Conditional Guardrails
participant Request as Request Intercepts
participant Sanitizers as Request / Response Sanitizers
participant Execution as Execution Intercepts
participant Callback as Real Callback
participant Dispatcher as Async Subscriber Dispatcher

Caller->>Runtime: managed tool or LLM call
Runtime->>Conditional: evaluate real request
alt rejected
Conditional-->>Runtime: rejection
Runtime-->>Caller: guardrail error
else allowed
Conditional-->>Runtime: continue
Runtime->>Request: transform real request
Request-->>Runtime: intercepted request
Runtime->>Sanitizers: sanitize event-only request copy
Sanitizers-->>Runtime: start-event payload
Runtime->>Dispatcher: enqueue start event
Runtime->>Execution: invoke intercept chain
alt intercept replaces execution
Execution-->>Runtime: replacement result
else intercept calls next
Execution->>Callback: invoke real callback
Callback-->>Execution: real result
Execution-->>Runtime: real result
end
Runtime->>Sanitizers: sanitize event-only response copy
Sanitizers-->>Runtime: end-event payload
Runtime->>Dispatcher: enqueue end event
Runtime-->>Caller: return real result
end
```

Two distinctions matter:

- Intercepts affect the real execution path
- Sanitize guardrails affect the emitted observability payload

For the expanded request-to-response runtime path, including streaming and subscriber handoff, refer to [Middleware](/about-nemo-relay/concepts/middleware#detailed-execution-flow).

## Runtime Layers

From bottom to top, NeMo Relay is organized as:

1. The Rust core runtime
2. The plugin and adaptive layer
3. Language bindings
4. Framework integrations and application code
5. Subscribers and observability backends

The details of a binding can vary, but the conceptual model stays the same across those layers.

## Design Goal

NeMo Relay is designed so that application developers, framework integrators, plugin authors, and observability consumers all reason about the same runtime semantics. One conceptual model should remain stable even when the binding or integration style changes.

## Related Concepts

The following concepts are related to this architecture:

- [Scopes](/about-nemo-relay/concepts/scopes)
- [Middleware](/about-nemo-relay/concepts/middleware)
- [Events](/about-nemo-relay/concepts/events)
- [Subscribers](/about-nemo-relay/concepts/subscribers)
- [Plugins](/about-nemo-relay/concepts/plugins)
- [Codecs](/about-nemo-relay/concepts/codecs)
For the expanded request-to-response runtime path, including streaming and
subscriber handoff, refer to
[Managed Execution Order](/about-nemo-relay/concepts/middleware#managed-execution-order).