Skip to content

docs: add concrete scope lifetime guidance #702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
afourniernv wants to merge 3 commits into
base: main
Choose a base branch
from
+65 −6
Open

docs: add concrete scope lifetime guidance #702

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e6faa4b
docs: add concrete scope lifetime guidance
afourniernv Aug 4, 2026
f288c16
docs: clarify scope terminology
afourniernv Aug 4, 2026
f31a49e
docs: correct the scope lifecycle example
afourniernv Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 65 additions & 6 deletions docs/about-nemo-relay/concepts/scopes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
---
title: "Scopes"
description: ""
description: "Understand scope hierarchy, lifetime, cleanup, and concurrent isolation."
position: 1
---
import { MermaidStyles } from "@/components/MermaidStyles";

{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0 */}

Expand All @@ -11,7 +13,7 @@ isolation.

## Why Scopes Exist

Scopes are the ownership backbone of NeMo Relay. Every tool call, LLM call, and
Scopes track where work belongs in NeMo Relay. Every tool call, LLM call, and
mark event attaches to a scope hierarchy.

That hierarchy lets the runtime:
Expand All @@ -30,14 +32,14 @@ A scope represents a logical unit of work such as:
- A request
- A workflow step
- A background task
- A nested function or tool orchestration boundary
- A nested function or tool workflow

Scopes are not just labels. They define ownership and visibility for other
runtime behavior.
Scopes are not just labels. They determine event parentage and which local
middleware and subscribers are visible.

## Scope Hierarchy and Ownership

Scopes form a tree. A child scope inherits the active execution context from its
Scopes form a tree. A child scope inherits the active context from its
parent and contributes new nested work beneath it.

That hierarchy determines:
Expand Down Expand Up @@ -90,6 +92,52 @@ and ends when it is popped or closed.
Scope-local middleware and subscribers are tied to the owning scope lifecycle.
When the scope closes, those registrations disappear automatically.

## Worked Scope Lifetime

Consider an agent that calls one tool and then one LLM:

1. The application or framework integration pushes an `Agent` scope beneath the
root and emits its start event.
2. The application registers middleware and a subscriber on the agent scope.
Both are visible while the agent or any of its nested scopes are active.
3. A managed tool wrapper emits the tool start event, runs the tool, and emits
the tool end event. The agent remains the active scope.
4. The application emits a mark under the active agent scope. The mark does not
change the stack.
5. A managed LLM wrapper emits the LLM start event, runs the model call, and
emits the LLM end event. The agent remains the active scope.
Comment on lines +103 to +108

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Show the nested Tool and Llm scope lifetimes.

The top-level Agent step is now correct. The tool and LLM steps still describe only event emission while the Agent remains active. The stack diagram also remains root → agent during both calls. This does not demonstrate the nested Tool and Llm lifetimes promised by the page. State that each managed wrapper pushes and pops its nested scope, and show root → agent → tool and root → agent → llm during the calls. This is the same unresolved lifecycle-actor issue from the previous review.

As per path instructions, documentation must be technically accurate against the current API and consistent across language bindings.

Also applies to: 123-126

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/about-nemo-relay/concepts/scopes.mdx` around lines 103 - 108, Update the
managed tool and LLM wrapper descriptions in the scope lifetime example to state
that each wrapper pushes its nested Tool or Llm scope before execution and pops
it afterward. Revise the corresponding stack diagrams to show root → agent →
tool during the tool call and root → agent → llm during the model call, while
preserving the agent scope as the parent.

Source: Path instructions

6. The application or framework integration emits the agent end event and pops
the agent scope. Its local
middleware and subscriber are removed.

The active scope stack changes over time. The emitted events remain in a
parent-linked tree after their scopes close. A mark event attaches beneath the
active scope but does not push another scope onto the stack.

<MermaidStyles />

```mermaid
flowchart LR
subgraph Stack["Active scope stack over time"]
direction LR
s1["root → agent<br/>agent starts"] --> s2["root → agent<br/>tool lifecycle emits"]
s2 --> s3["root → agent<br/>mark emits"]
s3 --> s4["root → agent<br/>LLM lifecycle emits"]
s4 --> s5["root<br/>agent closes"]
end

subgraph Events["Emitted event tree"]
direction TB
root["root"] --> agent["agent start/end"]
agent --> tool["tool start/end"]
agent --> mark["mark"]
agent --> llm["llm start/end"]
end
Comment thread
afourniernv marked this conversation as resolved.
```

Agent-local middleware and subscribers are visible during the first four stack
states. They are no longer visible after the agent scope closes.

## Semantic Payloads

Scopes may expose semantic `input` and `output` payloads on their emitted start
Expand All @@ -112,6 +160,17 @@ itself.
Context isolation keeps concurrent requests, tenants, and agents from sharing scope-
local state accidentally.

Choose the context behavior based on whether the work belongs to the same
logical trace and whether it runs concurrently:

| Work | Context Behavior |
| --- | --- |
| Sequential nested work in the same trace | Reuse the active stack |
| Concurrent branches in the same trace | Fork the stack for each branch |
| Independent work | Start with a fresh isolated stack |
| Same-trace work crossing a process boundary | Carry Relay propagation context; scope-local registrations do not cross the boundary |
| Independent work crossing a process boundary | Start with a fresh isolated stack instead of importing context |

### Why Isolation Matters

Concurrent requests must not share the same active scope stack accidentally.
Expand Down