Skip to content

feat(plugin): add Relay random and LLM-classifier routing - #220

Closed
bbednarski9 wants to merge 51 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feat/nemo-relay-dynamic-plugin
Closed

feat(plugin): add Relay random and LLM-classifier routing#220
bbednarski9 wants to merge 51 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feat/nemo-relay-dynamic-plugin

Conversation

@bbednarski9

@bbednarski9 bbednarski9 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What

Adds switchyard-nemo-relay-plugin, an external Relay native API v2 plugin that drives libsy::Algorithm::run_stream.

This PR intentionally supports exactly two router modes:

  • seeded, weighted random; and
  • capability-based llm_classifier, where a judge selects the weak or strong target before the final provider call.

stage_router and response-judging classifier escalation are rejected explicitly. They are isolated in draft follow-ups #250 and #251 so their request-mutation and response-buffering contracts can be reviewed independently.

For every managed call, the plugin decodes with switchyard-translation, consumes genuine Decision and CallLlm steps, invokes Relay's safe targeted LLM continuation, returns the real response or typed failure to libsy, and translates ReturnToAgent into the caller protocol. Relay owns HTTP transport, downstream LLM middleware, scopes, and observability. Switchyard owns routing, translation, targets, credentials, retry, and trusted fallback.

The plugin uses neither Relay codecs nor private dispatch headers, switchyard-llm-client, or switchyard-server. Unmanaged profiles use Relay-owned pass-through. Managed streams retry or fall back only before the first caller event. Routing marks use real libsy steps and exclude credentials and provider payloads.

The materialized schema tracks libsy's capability-classifier settings, including the current max_output_tokens verdict cap. The crate is a non-published cdylib build unit; operators will consume a binary bundle.

Why

A model-selection-only API cannot represent policies that issue LLM calls and inspect their responses. run_stream supplies the general algorithm lifecycle while leaving provider dispatch under Relay's control.

Prerequisites:

During review, nemo-relay-plugin is pinned to Relay revision 3a8f8f0745fc6545a9162bd585da2273fa052785. The Git pin will move to the corresponding official Relay commit, then to a compatible published SDK before binary bundles are released.

Relates to #192 and NVIDIA/NeMo-Relay#594.

How tested

  • uv run ruff check . clean
  • uv run mypy switchyard clean: 135 source files
  • Hermetic Python suite: 1,362 passed, 12 skipped, 24 live tests deselected
  • cargo test --workspace --all-targets passed, including switchyard-py, with one ignored live/VPN test
  • Manual smoke: real Relay process plus the three-protocol fake provider

Exact final-head validation at cda935c5360f85e5b87bb825fd51137c27183e0c, rebased onto Switchyard main at aac511923937eeaa4123adca21baf40a0ff5e821:

  • cargo test -p switchyard-nemo-relay-plugin: 19/19 passed;
  • cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check passed;
  • random: 27 decisions, 12 concurrent calls, and three provider protocols/models;
  • classifier: weak and strong routes, 6 concurrent calls, and OpenAI Chat, OpenAI Responses, and Anthropic Messages callers;
  • same- and cross-protocol buffered/streaming translation;
  • exact same-protocol unknown-field and raw-event preservation for all three protocols;
  • buffered and streaming retry reselection, exhaustion, and exactly-once fallback;
  • empty-stream fallback and committed late failure without retry;
  • credentials replaced at the target boundary and absent from recorded artifacts; and
  • unmanaged buffered/streaming pass-through with zero Switchyard marks.

The broader branch validation also passed workspace Rust/Python checks, docs, pre-commit, embedded-core routing without CLI gateway state, and local Ollama buffered/streaming smoke tests across two models with genuine Switchyard marks.

Checklist

  • One class per file; filename = snake_case of the primary class. (N/A: Rust crate.)
  • New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use. (N/A: no Python public symbols.)
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed.
  • Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

flowchart LR
    A["Caller request"] --> B["Relay LLM pipeline"]
    B --> C["Switchyard native API v2 callback"]
    C --> D["switchyard-translation decode"]
    D --> E["libsy run_stream"]
    E --> F["Decision / CallLlm"]
    F --> G["Relay targeted LLM continuation"]
    G --> H["Remaining Relay middleware"]
    H --> I["Relay core HTTP transport"]
    I --> J["Selected provider"]
    J --> G
    G --> K["CallLlmRequest.respond"]
    K --> E
    E --> L["ReturnToAgent"]
    L --> M["switchyard-translation encode"]
    M --> A
Loading

Please begin with crates/switchyard-nemo-relay-plugin/src/runtime.rs, then review config.rs, translation.rs, and tests/e2e/run_e2e.py.

Intentional constraints: native API v2 only; Random and capability classifier only; OpenAI Chat/Responses classifier judge targets only; no decision-only/observe-only mode; no synthesized telemetry; no retry/fallback after stream commitment; no binary release while the SDK remains Git-pinned.

This PR is ready for review but merge-dependent on #192 and NVIDIA/NeMo-Relay#594. Drafts #250 and #251 each branch from this exact head and do not include one another.

@bbednarski9 bbednarski9 changed the title feat(plugin): add NeMo Relay dynamic integration feat(plugin): add NeMo Relay run-stream dynamic plugin Aug 1, 2026
@bbednarski9
bbednarski9 force-pushed the feat/nemo-relay-dynamic-plugin branch 3 times, most recently from 4f23852 to 9659cd5 Compare August 3, 2026 04:54
@bbednarski9 bbednarski9 changed the title feat(plugin): add NeMo Relay run-stream dynamic plugin feat(plugin): add Relay random and LLM-classifier routing Aug 3, 2026
@bbednarski9
bbednarski9 marked this pull request as ready for review August 3, 2026 15:15
@bbednarski9
bbednarski9 requested a review from a team as a code owner August 3, 2026 15:15
@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Changes

Streaming and NeMo Relay integration

Layer / File(s) Summary
Preserved streaming event contract
crates/protocol/*, crates/switchyard-translation/*
Streaming items now carry normalized chunks and optional preserved provider events. Matching formats replay original events. Cross-format encoding uses normalized chunks.
Poll-driven libsy execution
crates/libsy/*
run_stream now runs when polled, converts panics to LibsyError::AlgorithmError, and cancels the producer future when dropped.
Plugin configuration and registration
crates/switchyard-nemo-relay-plugin/config.schema.json, crates/switchyard-nemo-relay-plugin/src/config.rs, crates/switchyard-nemo-relay-plugin/src/lib.rs, crates/switchyard-nemo-relay-plugin/relay-plugin.toml, crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py
The plugin adds version-two configuration, target validation, routing algorithms, dynamic-library metadata, bundle packaging, and native API registration.
Buffered and streaming routing
crates/switchyard-nemo-relay-plugin/src/runtime.rs, crates/switchyard-nemo-relay-plugin/src/translation.rs
The runtime adds request translation, provider dispatch, retries, fallback, terminal failure handling, routing marks, and late stream-error propagation.
End-to-end validation and documentation
crates/switchyard-nemo-relay-plugin/tests/e2e/*, README.md, docs/index.md
The test harness covers provider protocols, routing, replay, failures, passthrough, packaging, and credential handling. Documentation describes plugin use and installation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Poem

A rabbit watched the stream events flow,
With chunks preserved in neat rows.
Routes hopped fast, retries stayed bright,
Providers streamed through day and night.
“NeMo Relay is bundled right!”

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 56.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately identifies the new plugin and its supported random and LLM-classifier routing features.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (10)
crates/protocol/src/stream.rs (1)

222-239: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add a short comment for push_checked_chunk.

The helper encodes a non-obvious policy: an in-band DecodeError becomes ResponseTranslation, and an in-band StreamError becomes UpstreamHttp with MID_STREAM_UPSTREAM_STATUS. Record that mapping at the definition so the aggregation contract stays clear.

♻️ Proposed comment
+// Folds one chunk into the accumulator. In-band error chunks terminate aggregation with a
+// typed error instead of being folded: a decode failure is a translation error, and a provider
+// stream error is reported as a mid-stream upstream failure.
 fn push_checked_chunk(
     accumulator: &mut ResponseAccumulator,
     chunk: LlmResponseChunk,
 ) -> Result<(), LlmClientError> {

As per coding guidelines: "Add concise Rust documentation comments (///) for public items and comments for module intent, non-obvious private helpers, ...".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/protocol/src/stream.rs` around lines 222 - 239, Add a concise Rust
comment immediately above push_checked_chunk documenting its policy: DecodeError
maps to LlmClientError::ResponseTranslation, while StreamError maps to
LlmClientError::UpstreamHttp using MID_STREAM_UPSTREAM_STATUS; leave the
implementation unchanged.

Source: Coding guidelines

crates/switchyard-translation/tests/stream_translation.rs (1)

218-227: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Assert the delta by event type instead of a fixed index.

translated[2] depends on the exact number of Anthropic preamble events. If the encoder adds or reorders a preamble event, this test fails at an unrelated index and the cause is not obvious. Select the content_block_delta event instead.

♻️ Proposed change
     let preserved = engine.decode_stream_event(&mut state, WireFormat::OpenAiChat, event)?;
     let translated =
         engine.encode_stream_event(&mut state, WireFormat::AnthropicMessages, preserved)?;
 
-    assert_eq!(translated[2]["delta"]["text"], "Hi");
+    let delta = translated
+        .iter()
+        .find(|event| event["type"] == "content_block_delta")
+        .ok_or("the cross-format translation must emit a content_block_delta")?;
+    assert_eq!(delta["delta"]["text"], "Hi");
     assert!(
         translated
             .iter()
             .all(|event| event.get("system_fingerprint").is_none())
     );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-translation/tests/stream_translation.rs` around lines 218 -
227, Update the assertions in the stream translation test around
decode_stream_event and encode_stream_event to locate the event whose type is
content_block_delta, then assert its delta.text value is "Hi" instead of using
translated[2]. Keep the existing system_fingerprint assertion across all
translated events.
crates/switchyard-translation/src/helpers.rs (1)

209-214: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Avoid allocating a FormatId for every SSE frame.

FormatId owns a String, so source_format.clone() allocates and copies the identifier for each event. Consider a shared representation such as Arc<str>.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-translation/src/helpers.rs` around lines 209 - 214, Update
the event-stream path around LlmResponseStreamEvent::preserved to avoid cloning
source_format for each SSE frame. Change the shared format identifier
representation to Arc<str> (or reuse an existing shared representation), and
pass that value through preserved without per-event String allocation while
preserving the source format for every emitted event.
crates/switchyard-nemo-relay-plugin/src/config.rs (1)

14-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The new plugin crate omits the required Rust documentation comments. The crate-visible configuration types, the entry-point module, and the non-obvious URL rule carry no /// or //! comments. One documentation pass fixes all three sites.

  • crates/switchyard-nemo-relay-plugin/src/config.rs#L14-L14: add /// comments to protocol_from_call, PreparedTargetBinding, PreparedTargetBinding::dispatch_url, SwitchyardConfig, PreparedConfig, SwitchyardConfig::validate, and SwitchyardConfig::prepare, and state that validate performs static checks while prepare resolves environment-backed headers.
  • crates/switchyard-nemo-relay-plugin/src/config.rs#L46-L60: add a comment that states the /v1 de-duplication rule and its limits.
  • crates/switchyard-nemo-relay-plugin/src/lib.rs#L4-L17: add a //! module comment for the native API v2 entry point, and /// comments for SwitchyardPlugin and parse_config.

The coding guidelines require concise /// documentation for public items and comments for module intent and complex configuration logic.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/src/config.rs` at line 14, Add concise
Rust documentation across
crates/switchyard-nemo-relay-plugin/src/config.rs:14-14 for protocol_from_call,
PreparedTargetBinding, PreparedTargetBinding::dispatch_url, SwitchyardConfig,
PreparedConfig, SwitchyardConfig::validate, and SwitchyardConfig::prepare,
explicitly distinguishing static validation from environment-backed header
resolution; document crates/switchyard-nemo-relay-plugin/src/config.rs:46-60
with the /v1 de-duplication rule and its limits; and add a //! module-intent
comment plus /// comments for SwitchyardPlugin and parse_config in
crates/switchyard-nemo-relay-plugin/src/lib.rs:4-17.

Source: Coding guidelines

crates/switchyard-nemo-relay-plugin/config.schema.json (1)

34-63: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Declare the unsupported escalation field in the schema.

TaskClassifierConfig already contains every field listed in the llm_classifier branch. Add escalation with a description that states this plugin version does not support it. This allows the intended validation error instead of a generic oneOf failure.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/config.schema.json` around lines 34 - 63,
Add an optional escalation property to the llm_classifier schema object
alongside the existing TaskClassifierConfig fields, with a description
explicitly stating that this plugin version does not support escalation. Keep
its schema type aligned with the expected configuration value so validation
reports the intended unsupported-field error rather than a generic oneOf
failure.
crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py (1)

293-303: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document the deliberate content-length mismatch.

_sse_then_disconnect advertises len(data) + 64 bytes and then writes only len(data) bytes. The + 64 is intentional: it forces http.client.IncompleteRead in request_until_stream_error in run_e2e.py. A reader without that context sees a bug. Add a short docstring that states the intent.

📝 Proposed comment
     def _sse_then_disconnect(self, first: dict[str, object]) -> None:
+        """Emit one event, then truncate the body.
+
+        The advertised ``content-length`` exceeds the written bytes on purpose so
+        the client observes an incomplete read after response commitment.
+        """
         data = f"data: {json.dumps(first)}\n\n".encode()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py` around lines
293 - 303, Add a concise docstring to _sse_then_disconnect explaining that the
intentionally oversized content-length causes the client to receive an
incomplete response and triggers http.client.IncompleteRead in
request_until_stream_error. Preserve the existing +64 mismatch and disconnect
behavior.
crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py (2)

43-45: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Correct the http_json return annotation.

The annotation is dict[str, int]. Line 1026 calls http_json(provider_url, "/healthz"), and that endpoint returns {"ok": true}, which decodes to dict[str, bool]. The cast hides the mismatch from the type checker.

🛠️ Proposed fix
-def http_json(base: str, path: str) -> dict[str, int]:
+def http_json(base: str, path: str) -> dict[str, Any]:
     with urllib.request.urlopen(f"{base}{path}", timeout=5) as response:
-        return cast(dict[str, int], json.loads(response.read()))
+        return cast(dict[str, Any], json.loads(response.read()))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py` around lines 43 -
45, Update the http_json return annotation to reflect the JSON response shape,
allowing boolean values such as the {"ok": true} returned by the /healthz
endpoint; adjust the cast consistently so it no longer hides the dict[str, bool]
mismatch.

277-291: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Make the negative mark assertion explicit instead of using expected=0.

marks returns as soon as len(matches) >= expected. With expected=0 the condition is always true, so the method returns on the first read and never waits. The call sites at Line 674, Line 719, Line 764, Line 823, Line 824, Line 852, Line 877, and Lines 935-939 rely on that behavior to assert absence. The assertions are sound today because they run after __exit__ has waited for Relay to exit and flush, but the intent is not visible at the call site. Add a dedicated method so the two behaviors do not share one parameter.

♻️ Proposed refactor
     def marks(self, name: str, expected: int = 1, timeout: float = 5) -> list[dict[str, object]]:
         deadline = time.time() + timeout
         while True:
             try:
                 events = [
                     json.loads(line)
                     for line in self.atof_path.read_text(encoding="utf-8").splitlines()
                     if line
                 ]
             except FileNotFoundError:
                 events = []
             matches = [event for event in events if event.get("name") == name]
             if len(matches) >= expected or time.time() > deadline:
                 return matches
             time.sleep(0.05)
+
+    def no_marks(self, name: str) -> bool:
+        """Read the flushed sink once and report that no mark with ``name`` exists."""
+        return not self.marks(name, expected=0, timeout=0)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py` around lines 277 -
291, Update the test helper around marks to separate positive-count waiting from
negative mark assertions: add a dedicated method for asserting or waiting that
no event with the given name exists, then migrate the listed absence call sites
to it instead of passing expected=0. Keep marks focused on waiting for at least
the requested positive count and preserve the existing timeout behavior.
crates/switchyard-nemo-relay-plugin/src/translation.rs (1)

94-114: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add short doc comments for the policy helpers.

policy and request_policy encode non-obvious decisions. request_policy disables supports_json_schema_response_format only for WireFormat::AnthropicMessages, and the reason is not visible at the call site. Add /// comments that state the intent.

As per coding guidelines: "Add concise Rust documentation comments (///) for public items and comments for module intent, non-obvious private helpers, important tests, and complex validation, routing, configuration, async, lifecycle, or concurrency logic."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/src/translation.rs` around lines 94 -
114, Add concise Rust doc comments to the private helpers policy and
request_policy, documenting their translation-policy intent and the
request-specific behavior that disables supports_json_schema_response_format for
WireFormat::AnthropicMessages. Keep the implementation unchanged.

Source: Coding guidelines

crates/switchyard-nemo-relay-plugin/src/runtime.rs (1)

27-47: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add the missing Rust doc comments in the new plugin crate. Both new modules define non-obvious routing, streaming, and translation-policy logic without any /// comments. The shared root cause is that the crate ships no item documentation.

  • crates/switchyard-nemo-relay-plugin/src/runtime.rs#L27-L47: document SwitchyardRuntime, then document execute_buffered, execute_stream, and routed_stream, including the commitment rule and the fallback-once rule.
  • crates/switchyard-nemo-relay-plugin/src/translation.rs#L94-L114: document policy and request_policy, and state why supports_json_schema_response_format is disabled for WireFormat::AnthropicMessages.

As per coding guidelines: "Add concise Rust documentation comments (///) for public items and comments for module intent, non-obvious private helpers, important tests, and complex validation, routing, configuration, async, lifecycle, or concurrency logic."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-nemo-relay-plugin/src/runtime.rs` around lines 27 - 47, Add
concise Rust documentation in
crates/switchyard-nemo-relay-plugin/src/runtime.rs:27-47 for SwitchyardRuntime,
execute_buffered, execute_stream, and routed_stream, covering the commitment
rule and fallback-once behavior. In
crates/switchyard-nemo-relay-plugin/src/translation.rs:94-114, document policy
and request_policy, including why supports_json_schema_response_format is
disabled for WireFormat::AnthropicMessages. Also add module-intent or
non-obvious logic comments where needed, without changing behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-nemo-relay-plugin/Cargo.toml`:
- Line 22: Update the nemo-relay-plugin dependency in Cargo.toml to use the
upstream NVIDIA/NeMo-Relay repository URL, then regenerate Cargo.lock to reflect
the updated source. Keep the pinned revision while PR 594 remains open, and
switch to the published crate version after that PR merges.

In `@crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py`:
- Around line 48-51: Update the manifest substitution logic in the packaging
flow to validate that both <platform-library-file> and <artifact-sha256> are
present before replacing them. Fail clearly instead of writing the output
manifest when either placeholder is missing, while preserving the existing
substitutions when both are found.

In `@crates/switchyard-nemo-relay-plugin/src/runtime.rs`:
- Around line 262-266: Replace the panicking lookup in default_target with a
Result-returning lookup that reports a missing default target error. Update both
default_target call sites in the fallback paths to propagate this error with ?,
preserving their existing successful behavior.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py`:
- Around line 39-42: Update do_POST to validate that the request includes a
usable Content-Length before reading and parsing the body; reject missing or
chunked requests explicitly instead of defaulting to an empty JSON object and
model "unknown". Preserve normal parsing for requests with a valid body length.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py`:
- Around line 1054-1060: Update the credential scan in the recorded
comprehension to read candidate files with UTF-8 replacement handling by passing
errors="replace" to path.read_text. Preserve the existing suffix filtering,
TARGET_AUTHORIZATION search, and assertion behavior so undecodable files still
produce a credential verdict.

---

Nitpick comments:
In `@crates/protocol/src/stream.rs`:
- Around line 222-239: Add a concise Rust comment immediately above
push_checked_chunk documenting its policy: DecodeError maps to
LlmClientError::ResponseTranslation, while StreamError maps to
LlmClientError::UpstreamHttp using MID_STREAM_UPSTREAM_STATUS; leave the
implementation unchanged.

In `@crates/switchyard-nemo-relay-plugin/config.schema.json`:
- Around line 34-63: Add an optional escalation property to the llm_classifier
schema object alongside the existing TaskClassifierConfig fields, with a
description explicitly stating that this plugin version does not support
escalation. Keep its schema type aligned with the expected configuration value
so validation reports the intended unsupported-field error rather than a generic
oneOf failure.

In `@crates/switchyard-nemo-relay-plugin/src/config.rs`:
- Line 14: Add concise Rust documentation across
crates/switchyard-nemo-relay-plugin/src/config.rs:14-14 for protocol_from_call,
PreparedTargetBinding, PreparedTargetBinding::dispatch_url, SwitchyardConfig,
PreparedConfig, SwitchyardConfig::validate, and SwitchyardConfig::prepare,
explicitly distinguishing static validation from environment-backed header
resolution; document crates/switchyard-nemo-relay-plugin/src/config.rs:46-60
with the /v1 de-duplication rule and its limits; and add a //! module-intent
comment plus /// comments for SwitchyardPlugin and parse_config in
crates/switchyard-nemo-relay-plugin/src/lib.rs:4-17.

In `@crates/switchyard-nemo-relay-plugin/src/runtime.rs`:
- Around line 27-47: Add concise Rust documentation in
crates/switchyard-nemo-relay-plugin/src/runtime.rs:27-47 for SwitchyardRuntime,
execute_buffered, execute_stream, and routed_stream, covering the commitment
rule and fallback-once behavior. In
crates/switchyard-nemo-relay-plugin/src/translation.rs:94-114, document policy
and request_policy, including why supports_json_schema_response_format is
disabled for WireFormat::AnthropicMessages. Also add module-intent or
non-obvious logic comments where needed, without changing behavior.

In `@crates/switchyard-nemo-relay-plugin/src/translation.rs`:
- Around line 94-114: Add concise Rust doc comments to the private helpers
policy and request_policy, documenting their translation-policy intent and the
request-specific behavior that disables supports_json_schema_response_format for
WireFormat::AnthropicMessages. Keep the implementation unchanged.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py`:
- Around line 293-303: Add a concise docstring to _sse_then_disconnect
explaining that the intentionally oversized content-length causes the client to
receive an incomplete response and triggers http.client.IncompleteRead in
request_until_stream_error. Preserve the existing +64 mismatch and disconnect
behavior.

In `@crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py`:
- Around line 43-45: Update the http_json return annotation to reflect the JSON
response shape, allowing boolean values such as the {"ok": true} returned by the
/healthz endpoint; adjust the cast consistently so it no longer hides the
dict[str, bool] mismatch.
- Around line 277-291: Update the test helper around marks to separate
positive-count waiting from negative mark assertions: add a dedicated method for
asserting or waiting that no event with the given name exists, then migrate the
listed absence call sites to it instead of passing expected=0. Keep marks
focused on waiting for at least the requested positive count and preserve the
existing timeout behavior.

In `@crates/switchyard-translation/src/helpers.rs`:
- Around line 209-214: Update the event-stream path around
LlmResponseStreamEvent::preserved to avoid cloning source_format for each SSE
frame. Change the shared format identifier representation to Arc<str> (or reuse
an existing shared representation), and pass that value through preserved
without per-event String allocation while preserving the source format for every
emitted event.

In `@crates/switchyard-translation/tests/stream_translation.rs`:
- Around line 218-227: Update the assertions in the stream translation test
around decode_stream_event and encode_stream_event to locate the event whose
type is content_block_delta, then assert its delta.text value is "Hi" instead of
using translated[2]. Keep the existing system_fingerprint assertion across all
translated events.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fae968d1-9c44-4d99-bb57-7606cf972a68

📥 Commits

Reviewing files that changed from the base of the PR and between 3acf3d8 and ac937ce.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock, !Cargo.lock
📒 Files selected for processing (37)
  • Cargo.toml
  • README.md
  • crates/libsy/Cargo.toml
  • crates/libsy/examples/streaming_agent.rs
  • crates/libsy/src/algorithms/fall_through.rs
  • crates/libsy/src/algorithms/llm_class.rs
  • crates/libsy/src/algorithms/util/llm_judge.rs
  • crates/libsy/src/core/algorithm.rs
  • crates/libsy/src/core/driver.rs
  • crates/libsy/src/error.rs
  • crates/libsy/src/lib.rs
  • crates/libsy/src/observability.rs
  • crates/libsy/tests/observability.rs
  • crates/protocol/src/lib.rs
  • crates/protocol/src/stream.rs
  • crates/switchyard-nemo-relay-plugin/Cargo.toml
  • crates/switchyard-nemo-relay-plugin/README.md
  • crates/switchyard-nemo-relay-plugin/config.schema.json
  • crates/switchyard-nemo-relay-plugin/relay-plugin.toml
  • crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py
  • crates/switchyard-nemo-relay-plugin/src/config.rs
  • crates/switchyard-nemo-relay-plugin/src/lib.rs
  • crates/switchyard-nemo-relay-plugin/src/runtime.rs
  • crates/switchyard-nemo-relay-plugin/src/translation.rs
  • crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py
  • crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py
  • crates/switchyard-server/src/usage_metrics.rs
  • crates/switchyard-translation/src/codecs/anthropic/stream.rs
  • crates/switchyard-translation/src/codecs/openai_chat/stream.rs
  • crates/switchyard-translation/src/codecs/responses/stream.rs
  • crates/switchyard-translation/src/codecs/stream.rs
  • crates/switchyard-translation/src/engine.rs
  • crates/switchyard-translation/src/helpers.rs
  • crates/switchyard-translation/src/lib.rs
  • crates/switchyard-translation/tests/extension_points.rs
  • crates/switchyard-translation/tests/stream_translation.rs
  • docs/index.md

Comment thread crates/switchyard-nemo-relay-plugin/Cargo.toml Outdated
Comment thread crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py Outdated
Comment thread crates/switchyard-nemo-relay-plugin/src/runtime.rs Outdated
Comment thread crates/switchyard-nemo-relay-plugin/tests/e2e/fake_provider.py
Comment thread crates/switchyard-nemo-relay-plugin/tests/e2e/run_e2e.py
@bbednarski9
bbednarski9 force-pushed the feat/nemo-relay-dynamic-plugin branch 3 times, most recently from e2f501c to 1fb2bd5 Compare August 3, 2026 17:34
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…replay

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…al event

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Keep preservation on TranslationEngine, the API consumed by NVIDIA/NeMo-Relay#586, and remove the unused built-in convenience decoder.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant