From 9e65526f6a775ffe0934f1fda0baf02889c8bfa1 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 10:25:24 -0700 Subject: [PATCH 1/6] docs: update runtime architecture ownership Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 161 +++++++++++++------------ 1 file changed, 83 insertions(+), 78 deletions(-) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index 8af4a69d2..b0e2ae54f 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -8,96 +8,108 @@ import { MermaidStyles } from "@/components/MermaidStyles"; {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -This page explains how NeMo Relay connects scopes, middleware, plugins, events, -subscribers, and exporters. +This page explains how NeMo Relay connects [scopes](/about-nemo-relay/concepts/scopes), +[middleware](/about-nemo-relay/concepts/middleware), +[plugins](/about-nemo-relay/concepts/plugins), +[events](/about-nemo-relay/concepts/events), and +[subscribers](/about-nemo-relay/concepts/subscribers). ## Architecture Diagram -This diagram connects the runtime pieces to the layers they inhabit. +This diagram shows the CLI and language bindings hosting the shared Rust +runtime. Plugin components install behavior into that runtime, and subscribers +deliver event snapshots to in-process consumers or external backends. ```mermaid flowchart TB - subgraph AppLayer[Framework Integrations and Application Code] + subgraph Hosts[Runtime Hosts] + CLI[NeMo Relay CLI] App[Application Code] Framework[Framework Integration] + Bindings[Rust / Python / Node.js Bindings] + App -->|uses| Framework + App -. direct use .-> Bindings + Framework -->|calls| Bindings end - subgraph BindingLayer[Language Bindings] - Bindings[Language Bindings] - end + Config[Plugin Configuration] - subgraph PluginLayer[Plugin and Adaptive Layer] - PluginSystem[Plugin System] - Adaptive[Adaptive Component] - end - - subgraph CoreLayer[Core Runtime] + subgraph Runtime[Relay Runtime] Core[Rust Core Runtime] + PluginSystem[Plugin System] + Components[Installed Plugin Components] subgraph RuntimeState[Runtime State] - Scope[Scope Stack] + Scope[Active Scope Stack] Registry[Middleware Registries] + Events[Parent-Linked Event Records] end - Events[Event Stream] Dispatcher[Async Subscriber Dispatcher] + + PluginSystem -->|activates| Components + Components -->|register| Registry + Components -->|register| Dispatcher + Core -->|updates| Scope + Core -->|resolves| Registry + Core -->|emits| Events + Events -->|enqueue snapshots| Dispatcher end - subgraph ObsLayer[Subscribers and Observability Backends] - Subs[Subscribers / Exporters] - Backends[Files / OTLP / Other Backends] + subgraph Destinations[Subscriber and Export Destinations] + InProcess[In-Process Subscribers] + Files[ATOF / ATIF Files] + Backends[OTLP / HTTP / S3-Compatible Backends] end - App -->|uses| Framework - App -. direct use .-> Bindings - App -->|registers and configures| PluginSystem - Framework -->|calls| Bindings + CLI -->|hosts| Core Bindings --> Core - Adaptive -->|activates via| PluginSystem - PluginSystem -->|installs| Registry - PluginSystem -->|installs| Subs - Core -->|updates| Scope - Core -->|resolves| Registry - Core -->|emits| Events - Events -->|enqueue snapshots| Dispatcher - Dispatcher -->|deliver FIFO| Subs - Subs -->|export to| Backends - - class AppLayer grey-hint; - class BindingLayer grey-hint; - class PluginLayer grey-hint; - class CoreLayer grey-hint; - class ObsLayer grey-hint; + Config --> PluginSystem + Dispatcher -->|deliver FIFO| InProcess + Dispatcher -->|export| Files + Dispatcher -->|export| Backends + + class Hosts grey-hint; + class Runtime grey-hint; + class Destinations grey-hint; class RuntimeState grey-lightest; + class CLI purple-lightest; class App purple-lightest; class Framework yellow-lightest; class Bindings green-lightest; + class Config blue-lightest; class PluginSystem green-light; - class Adaptive blue-lightest; + class Components blue-lightest; class Core green-light; class Scope green-light; class Registry green-light; class Events green-light; - class Subs green-light; + class Dispatcher green-light; + class InProcess green-lightest; + class Files grey-light; class Backends grey-light; ``` -Adaptive appears here as a built-in plugin component rather than a separate runtime model because it activates through the same plugin lifecycle. +Adaptive behavior is one plugin component installed through the same lifecycle +as observability or PII redaction. It is not a separate runtime layer. ## Runtime Model NeMo Relay combines a small number of runtime pieces into one shared execution model: -- The **scope stack** answers where work belongs -- The **middleware registries** answer what should happen around that work -- The **plugin system** installs reusable runtime behavior from configuration -- The **event stream** records what happened +- The **scope stack** answers where work currently belongs. +- The **middleware registries** answer what should happen around that work. +- The **plugin system** installs reusable runtime behavior from configuration. +- The **event records** preserve what happened and how work was related. - The **async subscriber dispatcher** delivers event snapshots after emission -- **subscribers** consume those events +- **subscribers and exporters** consume those snapshots. -Every emitted scope, tool, LLM, or mark event attaches to the active scope stack. Every managed tool or LLM call resolves the currently visible middleware before it executes. +Every managed tool or LLM call resolves the middleware visible from the active +scope before it executes. When the runtime emits an event, it records the active +scope UUID as parentage. The scope stack changes as work opens and closes; the +parent-linked event records remain available to subscribers. ## Main Runtime Pieces @@ -105,7 +117,8 @@ These components are the primary building blocks that make up the runtime model. ### Scope Stack -The active scope stack defines the ownership tree for runtime work. It establishes: +The active scope stack defines the current ownership context for runtime work. +It establishes: - Parent-child relationships between events - Scope-local visibility for middleware and subscribers @@ -114,7 +127,15 @@ The active scope stack defines the ownership tree for runtime work. It establish ### Middleware Registries -The middleware registries hold the active intercepts and guardrails for tool and LLM execution. Managed helpers read those registries before invoking the real callback. +The middleware registries hold the active intercepts and guardrails for tool and +LLM execution. Request intercepts can rewrite real requests, conditional +guardrails can reject execution, and sanitize guardrails can change emitted +observability payloads. Managed helpers read those registries before invoking +the real callback. + +Async middleware callbacks are awaited as part of managed execution. The +managed call does not advance past that middleware checkpoint until the callback +returns. ### Plugin System @@ -122,13 +143,24 @@ The plugin system installs reusable runtime components from configuration. A plu ### Event Emission -The runtime emits structured events for scopes, tools, LLMs, and named marks. Those events are the canonical record of runtime behavior. Native Rust, Python, Node.js, and FFI event-producing APIs enqueue subscriber work and return without waiting for subscriber callbacks or exporter work. +The runtime emits structured events for scopes, tools, LLMs, and named marks. +Those parent-linked records are the canonical history of runtime behavior. +Native Rust, Python, Node.js, and FFI event-producing APIs enqueue subscriber +work and return without waiting for subscriber callbacks or exporter work. ### Subscribers and Exporters -Subscribers consume the event stream through the background dispatcher. Some subscribers stay in-process. Others export that stream into files or tracing systems. Use the binding flush API when a test or shutdown path must wait for already-queued subscriber work. +Subscribers consume event snapshots through the background dispatcher. Some +stay in process. Exporter subscribers can write +[ATOF JSONL](/configure-plugins/observability/atof), project events into +[ATIF trajectories](/configure-plugins/observability/atif), or emit +[OpenTelemetry traces](/configure-plugins/observability/opentelemetry). -## Two Axes of Runtime State +This delivery happens after event submission and is separate from awaited async +middleware. Use the binding flush API when a test or shutdown path must wait for +already queued subscriber work. + +## Where Runtime State Lives Runtime state is easiest to understand by separating ownership from process-wide registration. @@ -168,30 +200,3 @@ Two distinctions matter: - Sanitize guardrails affect the emitted observability payload For the expanded request-to-response runtime path, including streaming and subscriber handoff, refer to [Middleware](/about-nemo-relay/concepts/middleware#detailed-execution-flow). - -## Runtime Layers - -From bottom to top, NeMo Relay is organized as: - -1. The Rust core runtime -2. The plugin and adaptive layer -3. Language bindings -4. Framework integrations and application code -5. Subscribers and observability backends - -The details of a binding can vary, but the conceptual model stays the same across those layers. - -## Design Goal - -NeMo Relay is designed so that application developers, framework integrators, plugin authors, and observability consumers all reason about the same runtime semantics. One conceptual model should remain stable even when the binding or integration style changes. - -## Related Concepts - -The following concepts are related to this architecture: - -- [Scopes](/about-nemo-relay/concepts/scopes) -- [Middleware](/about-nemo-relay/concepts/middleware) -- [Events](/about-nemo-relay/concepts/events) -- [Subscribers](/about-nemo-relay/concepts/subscribers) -- [Plugins](/about-nemo-relay/concepts/plugins) -- [Codecs](/about-nemo-relay/concepts/codecs) From 4c47523f4a2142635a29a38fc7b82cc2d5441d77 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 13:19:54 -0700 Subject: [PATCH 2/6] docs: clarify runtime architecture Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index b0e2ae54f..5bb1bb78e 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -97,14 +97,15 @@ as observability or PII redaction. It is not a separate runtime layer. ## Runtime Model -NeMo Relay combines a small number of runtime pieces into one shared execution model: +NeMo Relay combines a small number of runtime pieces into one shared execution +model: - The **scope stack** answers where work currently belongs. - The **middleware registries** answer what should happen around that work. - The **plugin system** installs reusable runtime behavior from configuration. - The **event records** preserve what happened and how work was related. -- The **async subscriber dispatcher** delivers event snapshots after emission -- **subscribers and exporters** consume those snapshots. +- The **async subscriber dispatcher** delivers event snapshots after emission. +- **Subscribers and exporters** consume those snapshots. Every managed tool or LLM call resolves the middleware visible from the active scope before it executes. When the runtime emits an event, it records the active @@ -117,7 +118,7 @@ These components are the primary building blocks that make up the runtime model. ### Scope Stack -The active scope stack defines the current ownership context for runtime work. +The active scope stack defines the current context for runtime work. It establishes: - Parent-child relationships between events @@ -134,12 +135,14 @@ observability payloads. Managed helpers read those registries before invoking the real callback. Async middleware callbacks are awaited as part of managed execution. The -managed call does not advance past that middleware checkpoint until the callback +managed call does not advance past that middleware callback until the callback returns. ### Plugin System -The plugin system installs reusable runtime components from configuration. A plugin can register middleware, subscribers, or related behavior without requiring each application call site to do the work manually. +The plugin system installs reusable runtime components from configuration. A +plugin can register middleware, subscribers, or related behavior without +requiring each application call site to repeat the setup. ### Event Emission @@ -199,4 +202,6 @@ Two distinctions matter: - Intercepts affect the real execution path - Sanitize guardrails affect the emitted observability payload -For the expanded request-to-response runtime path, including streaming and subscriber handoff, refer to [Middleware](/about-nemo-relay/concepts/middleware#detailed-execution-flow). +For the expanded request-to-response runtime path, including streaming and +subscriber handoff, refer to +[Managed Execution Order](/about-nemo-relay/concepts/middleware#managed-execution-order). From 178c80f409e93a9bf13a05d609e4b691b70fc01a Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 13:53:25 -0700 Subject: [PATCH 3/6] docs: link runtime model concepts Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index 5bb1bb78e..8a7ffc2b2 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -100,12 +100,18 @@ as observability or PII redaction. It is not a separate runtime layer. NeMo Relay combines a small number of runtime pieces into one shared execution model: -- The **scope stack** answers where work currently belongs. -- The **middleware registries** answer what should happen around that work. -- The **plugin system** installs reusable runtime behavior from configuration. -- The **event records** preserve what happened and how work was related. -- The **async subscriber dispatcher** delivers event snapshots after emission. -- **Subscribers and exporters** consume those snapshots. +- The [**scope stack**](/about-nemo-relay/concepts/scopes) answers where work + currently belongs. +- The [**middleware registries**](/about-nemo-relay/concepts/middleware) answer + what should happen around that work. +- The [**plugin system**](/about-nemo-relay/concepts/plugins) installs reusable + runtime behavior from configuration. +- The [**event records**](/about-nemo-relay/concepts/events) preserve what + happened and how work was related. +- The [**async subscriber dispatcher**](/about-nemo-relay/concepts/subscribers#waiting-for-delivery) + delivers event snapshots after emission. +- [**Subscribers and exporters**](/about-nemo-relay/concepts/subscribers#common-subscriber-roles) + consume those snapshots. Every managed tool or LLM call resolves the middleware visible from the active scope before it executes. When the runtime emits an event, it records the active From 749e913697434eae9f7e2e72bbef66a4b9dd0ee2 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 14:05:02 -0700 Subject: [PATCH 4/6] docs: describe subscriber destinations Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index 8a7ffc2b2..714b0d811 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -164,6 +164,9 @@ stay in process. Exporter subscribers can write [ATOF JSONL](/configure-plugins/observability/atof), project events into [ATIF trajectories](/configure-plugins/observability/atif), or emit [OpenTelemetry traces](/configure-plugins/observability/opentelemetry). +Typical destinations include in-process application logic, local files and +artifact pipelines, OTLP-compatible observability backends, and evaluation or +visualization tools. This delivery happens after event submission and is separate from awaited async middleware. Use the binding flush API when a test or shutdown path must wait for From 326cb32bfe8998b74feef31568145cab0acd549f Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 14:50:54 -0700 Subject: [PATCH 5/6] docs: add runtime architecture diagrams Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 172 +++++++++++++++++++------ 1 file changed, 135 insertions(+), 37 deletions(-) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index 714b0d811..b8c540a6a 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -16,70 +16,73 @@ This page explains how NeMo Relay connects [scopes](/about-nemo-relay/concepts/s ## Architecture Diagram -This diagram shows the CLI and language bindings hosting the shared Rust -runtime. Plugin components install behavior into that runtime, and subscribers -deliver event snapshots to in-process consumers or external backends. +This diagram shows how runtime hosts and integrations reach the shared Rust +runtime. Plugin components install reusable behavior, and subscribers deliver +event snapshots to in-process consumers or external backends. ```mermaid flowchart TB - subgraph Hosts[Runtime Hosts] + subgraph AppLayer[Runtime Hosts and Integrations] CLI[NeMo Relay CLI] App[Application Code] Framework[Framework Integration] - Bindings[Rust / Python / Node.js Bindings] - App -->|uses| Framework - App -. direct use .-> Bindings - Framework -->|calls| Bindings end - Config[Plugin Configuration] + subgraph BindingLayer[Language Bindings] + Bindings[Language Bindings] + end - subgraph Runtime[Relay Runtime] - Core[Rust Core Runtime] + subgraph PluginLayer[Plugin System and Components] PluginSystem[Plugin System] - Components[Installed Plugin Components] + Components[Adaptive / Observability / Guardrail Components] + PluginSystem -->|activates| Components + end + + subgraph CoreLayer[Core Runtime] + Core[Rust Core Runtime] subgraph RuntimeState[Runtime State] - Scope[Active Scope Stack] + Scope[Scope Stack] Registry[Middleware Registries] - Events[Parent-Linked Event Records] end + Events[Event Stream] Dispatcher[Async Subscriber Dispatcher] - - PluginSystem -->|activates| Components - Components -->|register| Registry - Components -->|register| Dispatcher - Core -->|updates| Scope - Core -->|resolves| Registry - Core -->|emits| Events - Events -->|enqueue snapshots| Dispatcher end - subgraph Destinations[Subscriber and Export Destinations] - InProcess[In-Process Subscribers] - Files[ATOF / ATIF Files] - Backends[OTLP / HTTP / S3-Compatible Backends] + subgraph ObsLayer[Subscribers and Observability Backends] + Subs[Subscribers / Exporters] + Backends[Files / OTLP / Other Backends] end + App -->|uses| Framework + App -. direct use .-> Bindings + App -->|registers and configures| PluginSystem + Framework -->|calls| Bindings CLI -->|hosts| Core + CLI -->|loads configuration| PluginSystem Bindings --> Core - Config --> PluginSystem - Dispatcher -->|deliver FIFO| InProcess - Dispatcher -->|export| Files - Dispatcher -->|export| Backends - - class Hosts grey-hint; - class Runtime grey-hint; - class Destinations grey-hint; + Components -->|register middleware| Registry + Components -->|register subscribers| Subs + Core -->|updates| Scope + Core -->|resolves| Registry + Core -->|emits| Events + Events -->|enqueue snapshots| Dispatcher + Dispatcher -->|deliver FIFO| Subs + Subs -->|export to| Backends + + class AppLayer grey-hint; + class BindingLayer grey-hint; + class PluginLayer grey-hint; + class CoreLayer grey-hint; + class ObsLayer grey-hint; class RuntimeState grey-lightest; class CLI purple-lightest; class App purple-lightest; class Framework yellow-lightest; class Bindings green-lightest; - class Config blue-lightest; class PluginSystem green-light; class Components blue-lightest; class Core green-light; @@ -87,8 +90,7 @@ flowchart TB class Registry green-light; class Events green-light; class Dispatcher green-light; - class InProcess green-lightest; - class Files grey-light; + class Subs green-light; class Backends grey-light; ``` @@ -186,6 +188,29 @@ The scope stack defines: - When scope-local registrations are cleaned up - Whether concurrent requests stay isolated +Only scopes are pushed onto the stack. Managed LLM and tool calls emit lifecycle +records that use the current top scope as `parent_uuid`; they do not become +stack entries. This example shows an agent scope with a nested function scope +named `turn-a`. + +```mermaid +flowchart BT + Root["BOTTOM: Implicit root
uuid = root-a"] + Agent["Agent scope
uuid = agent-a
parent_uuid = root-a"] + Turn["TOP: Function scope (turn)
uuid = turn-a
parent_uuid = agent-a"] + + Root -->|"push agent-a"| Agent + Agent -->|"push turn-a"| Turn + + class Root grey-light; + class Agent,Turn green-light; +``` + +With `turn-a` at the top, managed LLM and tool records receive +`parent_uuid = turn-a`, so they are siblings in the event tree. Popping +`turn-a` returns `agent-a` to the top and removes registrations owned by the +turn. A concurrent request uses a separate stack. + ### Middleware Ownership Middleware exists at two levels: @@ -195,6 +220,40 @@ Middleware exists at two levels: That split lets long-lived defaults coexist with request-specific or task-specific behavior. +```mermaid +flowchart TB + Global["Global middleware
process-wide"] + + subgraph RequestA["Request A scope stack"] + LocalA["Scope-local middleware
owned by agent-a"] + ResolveA["Merge visible entries
and order by priority"] + CallA["Managed call A"] + CloseA["Close agent-a"] + LocalA --> ResolveA --> CallA + CloseA -.->|"removes"| LocalA + end + + subgraph RequestB["Request B scope stack"] + LocalB["Scope-local middleware
owned by agent-b"] + ResolveB["Merge visible entries
and order by priority"] + CallB["Managed call B"] + LocalB --> ResolveB --> CallB + end + + Global --> ResolveA + Global --> ResolveB + + class Global blue-lightest; + class RequestA,RequestB grey-lightest; + class LocalA,LocalB green-lightest; + class ResolveA,ResolveB green-light; + class CallA,CallB yellow-lightest; + class CloseA grey-light; +``` + +Global entries are visible to both calls. Each scope-local entry is visible +only through its owning stack and is removed when that scope closes. + ## Managed Execution Pipeline Managed tool and LLM execution follows the same high-level order: @@ -206,6 +265,45 @@ Managed tool and LLM execution follows the same high-level order: 5. The user callback runs. 6. Sanitize-response guardrails can rewrite the emitted end-event payload. +```mermaid +sequenceDiagram + autonumber + actor Caller as Application / Framework + participant Runtime as NeMo Relay Runtime + participant Conditional as Conditional Guardrails + participant Request as Request Intercepts + participant Sanitizers as Request / Response Sanitizers + participant Execution as Execution Intercepts + participant Callback as Real Callback + participant Dispatcher as Async Subscriber Dispatcher + + Caller->>Runtime: managed tool or LLM call + Runtime->>Conditional: evaluate real request + alt rejected + Conditional-->>Runtime: rejection + Runtime-->>Caller: guardrail error + else allowed + Conditional-->>Runtime: continue + Runtime->>Request: transform real request + Request-->>Runtime: intercepted request + Runtime->>Sanitizers: sanitize event-only request copy + Sanitizers-->>Runtime: start-event payload + Runtime->>Dispatcher: enqueue start event + Runtime->>Execution: invoke intercept chain + alt intercept replaces execution + Execution-->>Runtime: replacement result + else intercept calls next + Execution->>Callback: invoke real callback + Callback-->>Execution: real result + Execution-->>Runtime: real result + end + Runtime->>Sanitizers: sanitize event-only response copy + Sanitizers-->>Runtime: end-event payload + Runtime->>Dispatcher: enqueue end event + Runtime-->>Caller: return real result + end +``` + Two distinctions matter: - Intercepts affect the real execution path From e7aa708f3dac03c821e97e588a41aa980d6352df Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 16:18:22 -0700 Subject: [PATCH 6/6] docs: clarify runtime teardown Signed-off-by: Alex Fournier --- docs/about-nemo-relay/architecture.mdx | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/about-nemo-relay/architecture.mdx b/docs/about-nemo-relay/architecture.mdx index b8c540a6a..c45d4fb36 100644 --- a/docs/about-nemo-relay/architecture.mdx +++ b/docs/about-nemo-relay/architecture.mdx @@ -172,7 +172,15 @@ visualization tools. This delivery happens after event submission and is separate from awaited async middleware. Use the binding flush API when a test or shutdown path must wait for -already queued subscriber work. +already queued subscriber work. For manually registered exporters, follow the +exporter's documented teardown order before process exit. The +[ATOF](/configure-plugins/observability/atof), +[OpenTelemetry](/configure-plugins/observability/opentelemetry), and +[OpenInference](/configure-plugins/observability/openinference) exporters flush, +deregister, and then shut down. The +[ATIF](/configure-plugins/observability/atif) exporter drains subscriber work, +exports, deregisters, and then clears its state. Clearing plugin configuration +owns teardown for plugin-installed exporters. ## Where Runtime State Lives @@ -220,6 +228,9 @@ Middleware exists at two levels: That split lets long-lived defaults coexist with request-specific or task-specific behavior. +The following diagram shows how global and scope-local middleware are resolved +for two concurrent requests. + ```mermaid flowchart TB Global["Global middleware
process-wide"] @@ -265,6 +276,9 @@ Managed tool and LLM execution follows the same high-level order: 5. The user callback runs. 6. Sanitize-response guardrails can rewrite the emitted end-event payload. +The following sequence shows where each middleware family runs during a managed +call. + ```mermaid sequenceDiagram autonumber