diff --git a/docs/about-nemo-relay/concepts/scopes.mdx b/docs/about-nemo-relay/concepts/scopes.mdx index ecdb7ea46..a2e1b65e7 100644 --- a/docs/about-nemo-relay/concepts/scopes.mdx +++ b/docs/about-nemo-relay/concepts/scopes.mdx @@ -1,8 +1,10 @@ --- title: "Scopes" -description: "" +description: "Understand scope hierarchy, lifetime, cleanup, and concurrent isolation." position: 1 --- +import { MermaidStyles } from "@/components/MermaidStyles"; + {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} @@ -11,7 +13,7 @@ isolation. ## Why Scopes Exist -Scopes are the ownership backbone of NeMo Relay. Every tool call, LLM call, and +Scopes track where work belongs in NeMo Relay. Every tool call, LLM call, and mark event attaches to a scope hierarchy. That hierarchy lets the runtime: @@ -30,14 +32,14 @@ A scope represents a logical unit of work such as: - A request - A workflow step - A background task -- A nested function or tool orchestration boundary +- A nested function or tool workflow -Scopes are not just labels. They define ownership and visibility for other -runtime behavior. +Scopes are not just labels. They determine event parentage and which local +middleware and subscribers are visible. ## Scope Hierarchy and Ownership -Scopes form a tree. A child scope inherits the active execution context from its +Scopes form a tree. A child scope inherits the active context from its parent and contributes new nested work beneath it. That hierarchy determines: @@ -90,6 +92,52 @@ and ends when it is popped or closed. Scope-local middleware and subscribers are tied to the owning scope lifecycle. When the scope closes, those registrations disappear automatically. +## Worked Scope Lifetime + +Consider an agent that calls one tool and then one LLM: + +1. The application or framework integration pushes an `Agent` scope beneath the + root and emits its start event. +2. The application registers middleware and a subscriber on the agent scope. + Both are visible while the agent or any of its nested scopes are active. +3. A managed tool wrapper emits the tool start event, runs the tool, and emits + the tool end event. The agent remains the active scope. +4. The application emits a mark under the active agent scope. The mark does not + change the stack. +5. A managed LLM wrapper emits the LLM start event, runs the model call, and + emits the LLM end event. The agent remains the active scope. +6. The application or framework integration emits the agent end event and pops + the agent scope. Its local + middleware and subscriber are removed. + +The active scope stack changes over time. The emitted events remain in a +parent-linked tree after their scopes close. A mark event attaches beneath the +active scope but does not push another scope onto the stack. + + + +```mermaid +flowchart LR + subgraph Stack["Active scope stack over time"] + direction LR + s1["root → agent
agent starts"] --> s2["root → agent
tool lifecycle emits"] + s2 --> s3["root → agent
mark emits"] + s3 --> s4["root → agent
LLM lifecycle emits"] + s4 --> s5["root
agent closes"] + end + + subgraph Events["Emitted event tree"] + direction TB + root["root"] --> agent["agent start/end"] + agent --> tool["tool start/end"] + agent --> mark["mark"] + agent --> llm["llm start/end"] + end +``` + +Agent-local middleware and subscribers are visible during the first four stack +states. They are no longer visible after the agent scope closes. + ## Semantic Payloads Scopes may expose semantic `input` and `output` payloads on their emitted start @@ -112,6 +160,17 @@ itself. Context isolation keeps concurrent requests, tenants, and agents from sharing scope- local state accidentally. +Choose the context behavior based on whether the work belongs to the same +logical trace and whether it runs concurrently: + +| Work | Context Behavior | +| --- | --- | +| Sequential nested work in the same trace | Reuse the active stack | +| Concurrent branches in the same trace | Fork the stack for each branch | +| Independent work | Start with a fresh isolated stack | +| Same-trace work crossing a process boundary | Carry Relay propagation context; scope-local registrations do not cross the boundary | +| Independent work crossing a process boundary | Start with a fresh isolated stack instead of importing context | + ### Why Isolation Matters Concurrent requests must not share the same active scope stack accidentally.