From e6faa4b347b147cefcc6e04812abb2ca7c00f7af Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 10:35:34 -0700 Subject: [PATCH 1/3] docs: add concrete scope lifetime guidance Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/scopes.mdx | 53 +++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/about-nemo-relay/concepts/scopes.mdx b/docs/about-nemo-relay/concepts/scopes.mdx index ecdb7ea46..aa0a8d442 100644 --- a/docs/about-nemo-relay/concepts/scopes.mdx +++ b/docs/about-nemo-relay/concepts/scopes.mdx @@ -3,6 +3,8 @@ title: "Scopes" description: "" position: 1 --- +import { MermaidStyles } from "@/components/MermaidStyles"; + {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} @@ -90,6 +92,47 @@ and ends when it is popped or closed. Scope-local middleware and subscribers are tied to the owning scope lifecycle. When the scope closes, those registrations disappear automatically. +## Worked Scope Lifetime + +Consider an agent that calls one tool and then one LLM: + +1. Relay pushes an `Agent` scope beneath the root and emits its start event. +2. The application registers middleware and a subscriber on the agent scope. + Both are visible while the agent or any of its nested scopes are active. +3. Relay pushes a `Tool` scope, runs the tool, emits the tool events, and pops + the scope. The agent scope becomes active again. +4. Relay pushes an `Llm` scope, runs the model call, emits the LLM events, and + pops the scope. +5. Relay emits the agent end event and pops the agent scope. Its local + middleware and subscriber are removed. + +The active scope stack changes over time. The emitted events remain in a +parent-linked tree after their scopes close. A mark event attaches beneath the +active scope but does not push another scope onto the stack. + + + +```mermaid +flowchart LR + subgraph Stack["Active scope stack over time"] + direction LR + s1["root → agent"] --> s2["root → agent → tool"] + s2 --> s3["root → agent"] + s3 --> s4["root → agent → llm"] + s4 --> s5["root"] + end + + subgraph Events["Emitted event tree"] + direction TB + root["root"] --> agent["agent start/end"] + agent --> tool["tool start/end"] + agent --> llm["llm start/end"] + end +``` + +Agent-local middleware and subscribers are visible during the first four stack +states. They are no longer visible after the agent scope closes. + ## Semantic Payloads Scopes may expose semantic `input` and `output` payloads on their emitted start @@ -112,6 +155,16 @@ itself. Context isolation keeps concurrent requests, tenants, and agents from sharing scope- local state accidentally. +Choose the context behavior based on whether the work belongs to the same +logical trace and whether it runs concurrently: + +| Work | Context behavior | +| --- | --- | +| Sequential nested work in the same trace | Reuse the active stack | +| Concurrent branches in the same trace | Fork the stack for each branch | +| Independent work | Start with a fresh isolated stack | +| Work crossing a process boundary | Carry Relay propagation context | + ### Why Isolation Matters Concurrent requests must not share the same active scope stack accidentally. From f288c1611a113691b8f97eeb01c6741ed165be9a Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 13:19:54 -0700 Subject: [PATCH 2/3] docs: clarify scope terminology Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/scopes.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/about-nemo-relay/concepts/scopes.mdx b/docs/about-nemo-relay/concepts/scopes.mdx index aa0a8d442..7d72396e4 100644 --- a/docs/about-nemo-relay/concepts/scopes.mdx +++ b/docs/about-nemo-relay/concepts/scopes.mdx @@ -1,6 +1,6 @@ --- title: "Scopes" -description: "" +description: "Understand scope hierarchy, lifetime, cleanup, and concurrent isolation." position: 1 --- import { MermaidStyles } from "@/components/MermaidStyles"; @@ -13,7 +13,7 @@ isolation. ## Why Scopes Exist -Scopes are the ownership backbone of NeMo Relay. Every tool call, LLM call, and +Scopes track where work belongs in NeMo Relay. Every tool call, LLM call, and mark event attaches to a scope hierarchy. That hierarchy lets the runtime: @@ -32,14 +32,14 @@ A scope represents a logical unit of work such as: - A request - A workflow step - A background task -- A nested function or tool orchestration boundary +- A nested function or tool workflow -Scopes are not just labels. They define ownership and visibility for other -runtime behavior. +Scopes are not just labels. They determine event parentage and which local +middleware and subscribers are visible. ## Scope Hierarchy and Ownership -Scopes form a tree. A child scope inherits the active execution context from its +Scopes form a tree. A child scope inherits the active context from its parent and contributes new nested work beneath it. That hierarchy determines: From f31a49e59ddd53fdaf42ed3c9f6f2fc9a6785189 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 4 Aug 2026 16:18:27 -0700 Subject: [PATCH 3/3] docs: correct the scope lifecycle example Signed-off-by: Alex Fournier --- docs/about-nemo-relay/concepts/scopes.mdx | 30 ++++++++++++++--------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/about-nemo-relay/concepts/scopes.mdx b/docs/about-nemo-relay/concepts/scopes.mdx index 7d72396e4..a2e1b65e7 100644 --- a/docs/about-nemo-relay/concepts/scopes.mdx +++ b/docs/about-nemo-relay/concepts/scopes.mdx @@ -96,14 +96,18 @@ When the scope closes, those registrations disappear automatically. Consider an agent that calls one tool and then one LLM: -1. Relay pushes an `Agent` scope beneath the root and emits its start event. +1. The application or framework integration pushes an `Agent` scope beneath the + root and emits its start event. 2. The application registers middleware and a subscriber on the agent scope. Both are visible while the agent or any of its nested scopes are active. -3. Relay pushes a `Tool` scope, runs the tool, emits the tool events, and pops - the scope. The agent scope becomes active again. -4. Relay pushes an `Llm` scope, runs the model call, emits the LLM events, and - pops the scope. -5. Relay emits the agent end event and pops the agent scope. Its local +3. A managed tool wrapper emits the tool start event, runs the tool, and emits + the tool end event. The agent remains the active scope. +4. The application emits a mark under the active agent scope. The mark does not + change the stack. +5. A managed LLM wrapper emits the LLM start event, runs the model call, and + emits the LLM end event. The agent remains the active scope. +6. The application or framework integration emits the agent end event and pops + the agent scope. Its local middleware and subscriber are removed. The active scope stack changes over time. The emitted events remain in a @@ -116,16 +120,17 @@ active scope but does not push another scope onto the stack. flowchart LR subgraph Stack["Active scope stack over time"] direction LR - s1["root → agent"] --> s2["root → agent → tool"] - s2 --> s3["root → agent"] - s3 --> s4["root → agent → llm"] - s4 --> s5["root"] + s1["root → agent
agent starts"] --> s2["root → agent
tool lifecycle emits"] + s2 --> s3["root → agent
mark emits"] + s3 --> s4["root → agent
LLM lifecycle emits"] + s4 --> s5["root
agent closes"] end subgraph Events["Emitted event tree"] direction TB root["root"] --> agent["agent start/end"] agent --> tool["tool start/end"] + agent --> mark["mark"] agent --> llm["llm start/end"] end ``` @@ -158,12 +163,13 @@ local state accidentally. Choose the context behavior based on whether the work belongs to the same logical trace and whether it runs concurrently: -| Work | Context behavior | +| Work | Context Behavior | | --- | --- | | Sequential nested work in the same trace | Reuse the active stack | | Concurrent branches in the same trace | Fork the stack for each branch | | Independent work | Start with a fresh isolated stack | -| Work crossing a process boundary | Carry Relay propagation context | +| Same-trace work crossing a process boundary | Carry Relay propagation context; scope-local registrations do not cross the boundary | +| Independent work crossing a process boundary | Start with a fresh isolated stack instead of importing context | ### Why Isolation Matters