(MOT-4412) fix(llm-router): harden provider lifecycle - #812
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (25)
📝 WalkthroughWalkthroughChangesRouter reliability and integration coverage
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested reviewers: Poem
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 60 skipped (no docs/).
Four for four. Nicely done. |
ytallo
marked this pull request as ready for review
August 15, 2026 13:04
andersonleal
added a commit
that referenced
this pull request
Aug 17, 2026
…injection via configuration entries, hot-applied Move each worker's system-prompt usage guidance behind an inject_guidance knob (ON by default) in its builtin-configuration entry: flipping it binds or unbinds the pre-generate hook live, no restart, shrinking agent prompts when the guidance is not wanted. The non-harness half of the token-usage work. The config plumbing all five share lives in the new crates/config-client (iii-sdk range dep, the console-ui precedent): retry ladder with a NOT_FOUND fast-path, seed-only-when-nothing-stored (configuration::register REPLACES the stored value whenever initial_value is supplied, so the pre-check is load-bearing), case-SENSITIVE NOT_FOUND so an absent config plane never reads as "nothing stored yet", serialized reloads with the fetch inside the lock, and a post-bind boot refresh closing the fetch->bind gap; scrapling mirrors the same semantics in Python. All five treat the config path as best-effort at boot — warn and run on defaults rather than taking the worker's real surface off the bus (docs/sops/configuration.md now documents the cosmetic-knob exception). Also in this change: - sandbox-code-runner seeds on-config-change + ui-content into the claims registry (seeded_ids, the code-runner pattern), closing a boot window where a guest register_function could claim a late-registered worker id and abort the process via the SDK's duplicate-id panic. - sandbox-code-runner's dead custom console form is removed (it predated the entry and would have hidden the knob behind stale timeout fields); the console's schema-generated form renders the entry, and the README documents it. - fp ships no injected UI for its one boolean; the fp/ui package, build.rs, and src/ui.rs are gone and the schema form serves the knob. - llm-router resolves composite "provider::model" ids (the console display form) at the choke points: catalog queries retry an exact miss via the split pair with supports delegating to get, and chat/route/count_tokens split known-provider composites before dispatch — metadata and routing can never disagree about the same id. Rebased over the provider-lifecycle hardening (#812): availability checks run against the split pair. - workflow's stamp-reply and inject-guidance hook responses are typed structs (the interface publish gate refuses AnyValue response schemas); inject-guidance also adopts fp's rule of preserving the harness prompt on an empty/drifted base instead of replacing it with guidance alone, and stamp-reply's no-op answers an explicit continue (parsed identically to the old null). - provider-llamacpp, github-copilot, kimi, and openrouter tag their router-ready handlers internal, keeping the default engine::functions::list free of provider plumbing; the four providers' lockfiles are regenerated so the per-worker --locked gates resolve. - rust-security-audit audits every changed lockfile with a full fetch: the old --no-fetch on later iterations made each lockfile after the first fail its yanked lookups against a half-warmed index. - '!<worker>::on-config-change' denies for all five workers (web and workflow were missing theirs too) and configuration dependencies in the fp / sandbox-code-runner / workflow / scrapling manifests.
andersonleal
added a commit
that referenced
this pull request
Aug 17, 2026
…injection via configuration entries, hot-applied Move each worker's system-prompt usage guidance behind an inject_guidance knob (ON by default) in its builtin-configuration entry: flipping it binds or unbinds the pre-generate hook live, no restart, shrinking agent prompts when the guidance is not wanted. The non-harness half of the token-usage work. The config plumbing all five share lives in the new crates/config-client (iii-sdk range dep, the console-ui precedent): retry ladder with a NOT_FOUND fast-path, seed-only-when-nothing-stored (configuration::register REPLACES the stored value whenever initial_value is supplied, so the pre-check is load-bearing), case-SENSITIVE NOT_FOUND so an absent config plane never reads as "nothing stored yet", serialized reloads with the fetch inside the lock, and a post-bind boot refresh closing the fetch->bind gap; scrapling mirrors the same semantics in Python. All five treat the config path as best-effort at boot — warn and run on defaults rather than taking the worker's real surface off the bus (docs/sops/configuration.md now documents the cosmetic-knob exception). Also in this change: - sandbox-code-runner seeds on-config-change + ui-content into the claims registry (seeded_ids, the code-runner pattern), closing a boot window where a guest register_function could claim a late-registered worker id and abort the process via the SDK's duplicate-id panic. - sandbox-code-runner's dead custom console form is removed (it predated the entry and would have hidden the knob behind stale timeout fields); the console's schema-generated form renders the entry, and the README documents it. - fp ships no injected UI for its one boolean; the fp/ui package, build.rs, and src/ui.rs are gone and the schema form serves the knob. - llm-router resolves composite "provider::model" ids (the console display form) at the choke points: catalog queries retry an exact miss via the split pair with supports delegating to get, and chat/route/count_tokens split known-provider composites before dispatch — metadata and routing can never disagree about the same id. Rebased over the provider-lifecycle hardening (#812): availability checks run against the split pair. - workflow's stamp-reply and inject-guidance hook responses are typed structs (the interface publish gate refuses AnyValue response schemas); inject-guidance also adopts fp's rule of preserving the harness prompt on an empty/drifted base instead of replacing it with guidance alone, and stamp-reply's no-op answers an explicit continue (parsed identically to the old null). - provider-llamacpp, github-copilot, kimi, and openrouter tag their router-ready handlers internal, keeping the default engine::functions::list free of provider plumbing; the four providers' lockfiles are regenerated so the per-worker --locked gates resolve. - rust-security-audit audits every changed lockfile with a full fetch: the old --no-fetch on later iterations made each lockfile after the first fail its yanked lookups against a half-warmed index; the workflow-convention test pinning the old flag is updated, and the audit's first real catch on these lockfiles — quinn-proto RUSTSEC-2026-0185 in the kimi and web locks — is patched by a lock-only bump. - '!<worker>::on-config-change' denies for all five workers (web and workflow were missing theirs too) and configuration dependencies in the fp / sandbox-code-runner / workflow / scrapling manifests.
andersonleal
added a commit
that referenced
this pull request
Aug 18, 2026
…injection via configuration entries, hot-applied Move each worker's system-prompt usage guidance behind an inject_guidance knob (ON by default) in its builtin-configuration entry: flipping it binds or unbinds the pre-generate hook live, no restart, shrinking agent prompts when the guidance is not wanted. The non-harness half of the token-usage work. The config plumbing all five share lives in the new crates/config-client (iii-sdk range dep, the console-ui precedent): retry ladder with a NOT_FOUND fast-path, seed-only-when-nothing-stored (configuration::register REPLACES the stored value whenever initial_value is supplied, so the pre-check is load-bearing), case-SENSITIVE NOT_FOUND so an absent config plane never reads as "nothing stored yet", serialized reloads with the fetch inside the lock, and a post-bind boot refresh closing the fetch->bind gap; scrapling mirrors the same semantics in Python. All five treat the config path as best-effort at boot — warn and run on defaults rather than taking the worker's real surface off the bus (docs/sops/configuration.md now documents the cosmetic-knob exception). Also in this change: - sandbox-code-runner seeds on-config-change + ui-content into the claims registry (seeded_ids, the code-runner pattern), closing a boot window where a guest register_function could claim a late-registered worker id and abort the process via the SDK's duplicate-id panic. - sandbox-code-runner's dead custom console form is removed (it predated the entry and would have hidden the knob behind stale timeout fields); the console's schema-generated form renders the entry, and the README documents it. - fp ships no injected UI for its one boolean; the fp/ui package, build.rs, and src/ui.rs are gone and the schema form serves the knob. - llm-router resolves composite "provider::model" ids (the console display form) at the choke points: catalog queries retry an exact miss via the split pair with supports delegating to get, and chat/route/count_tokens split known-provider composites before dispatch — metadata and routing can never disagree about the same id. Rebased over the provider-lifecycle hardening (#812): availability checks run against the split pair. - workflow's stamp-reply and inject-guidance hook responses are typed structs (the interface publish gate refuses AnyValue response schemas); inject-guidance also adopts fp's rule of preserving the harness prompt on an empty/drifted base instead of replacing it with guidance alone, and stamp-reply's no-op answers an explicit continue (parsed identically to the old null). - provider-llamacpp, github-copilot, kimi, and openrouter tag their router-ready handlers internal, keeping the default engine::functions::list free of provider plumbing; the four providers' lockfiles are regenerated so the per-worker --locked gates resolve. - rust-security-audit audits every changed lockfile with a full fetch: the old --no-fetch on later iterations made each lockfile after the first fail its yanked lookups against a half-warmed index; the workflow-convention test pinning the old flag is updated, and the audit's first real catch on these lockfiles — quinn-proto RUSTSEC-2026-0185 in the kimi and web locks — is patched by a lock-only bump. - '!<worker>::on-config-change' denies for all five workers (web and workflow were missing theirs too) and configuration dependencies in the fp / sandbox-code-runner / workflow / scrapling manifests.
andersonleal
added a commit
that referenced
this pull request
Aug 18, 2026
…n guidance injection via configuration entries, hot-applied (#822) * feat(fp,web,workflow,sandbox-code-runner,scrapling): opt-in guidance injection via configuration entries, hot-applied Move each worker's system-prompt usage guidance behind an inject_guidance knob (ON by default) in its builtin-configuration entry: flipping it binds or unbinds the pre-generate hook live, no restart, shrinking agent prompts when the guidance is not wanted. The non-harness half of the token-usage work. The config plumbing all five share lives in the new crates/config-client (iii-sdk range dep, the console-ui precedent): retry ladder with a NOT_FOUND fast-path, seed-only-when-nothing-stored (configuration::register REPLACES the stored value whenever initial_value is supplied, so the pre-check is load-bearing), case-SENSITIVE NOT_FOUND so an absent config plane never reads as "nothing stored yet", serialized reloads with the fetch inside the lock, and a post-bind boot refresh closing the fetch->bind gap; scrapling mirrors the same semantics in Python. All five treat the config path as best-effort at boot — warn and run on defaults rather than taking the worker's real surface off the bus (docs/sops/configuration.md now documents the cosmetic-knob exception). Also in this change: - sandbox-code-runner seeds on-config-change + ui-content into the claims registry (seeded_ids, the code-runner pattern), closing a boot window where a guest register_function could claim a late-registered worker id and abort the process via the SDK's duplicate-id panic. - sandbox-code-runner's dead custom console form is removed (it predated the entry and would have hidden the knob behind stale timeout fields); the console's schema-generated form renders the entry, and the README documents it. - fp ships no injected UI for its one boolean; the fp/ui package, build.rs, and src/ui.rs are gone and the schema form serves the knob. - llm-router resolves composite "provider::model" ids (the console display form) at the choke points: catalog queries retry an exact miss via the split pair with supports delegating to get, and chat/route/count_tokens split known-provider composites before dispatch — metadata and routing can never disagree about the same id. Rebased over the provider-lifecycle hardening (#812): availability checks run against the split pair. - workflow's stamp-reply and inject-guidance hook responses are typed structs (the interface publish gate refuses AnyValue response schemas); inject-guidance also adopts fp's rule of preserving the harness prompt on an empty/drifted base instead of replacing it with guidance alone, and stamp-reply's no-op answers an explicit continue (parsed identically to the old null). - provider-llamacpp, github-copilot, kimi, and openrouter tag their router-ready handlers internal, keeping the default engine::functions::list free of provider plumbing; the four providers' lockfiles are regenerated so the per-worker --locked gates resolve. - rust-security-audit audits every changed lockfile with a full fetch: the old --no-fetch on later iterations made each lockfile after the first fail its yanked lookups against a half-warmed index; the workflow-convention test pinning the old flag is updated, and the audit's first real catch on these lockfiles — quinn-proto RUSTSEC-2026-0185 in the kimi and web locks — is patched by a lock-only bump. - '!<worker>::on-config-change' denies for all five workers (web and workflow were missing theirs too) and configuration dependencies in the fp / sandbox-code-runner / workflow / scrapling manifests. * fix(ci): bump h2 to 0.4.16 (RUSTSEC-2026-0258) and sync provider testkit lockfile with post-release provider versions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
INT-021, which streams partial content and keepalives before a permanent terminal provider errorRoot cause
Registry/catalog state could be published partially, stale provider responses could overwrite a newer availability generation, and duplicate request IDs could replace active cancellation state. Some pre-stream and bus failures returned without the terminal frame required by the router contract. Provider trigger cleanup also needed bounded admission without cancelling the SDK future before it could release pending state. The regular Rust CI job did not provide an engine, so live lifecycle coverage could silently skip.
Impact
Provider failure and recovery are deterministic: unavailable or stale providers are excluded from routing, registrations do not expose partial state, duplicate requests cannot orphan active streams, and consumers receive a terminal result instead of hanging. The live-engine CI lane now exercises these guarantees on every relevant change.
The Harness adversarial scenario verifies the complete queue, session, context, state, and harness path. It requires the failed run to preserve partial content, expose the exact terminal error, leave no pending function calls, avoid transient resume, and invoke the router exactly once.
Validation
python3 -m pytest .github/scripts/tests -q— 192 passed, 3 subtests passedcargo test --locked --manifest-path llm-router/Cargo.toml --lib --all-features -- --test-threads=1— 127 passedIII_ENGINE_BIN=/home/layon/.local/bin/iii cargo test --locked --manifest-path llm-router/Cargo.toml --no-default-features --test integration -- --nocapture --test-threads=1with iii 0.22.1 — 22 passedcargo test --locked --manifest-path llm-router/Cargo.toml --all-features --bins --test schemas -- --test-threads=1— 2 binary and 4 schema tests passedcargo clippy --locked --manifest-path llm-router/Cargo.toml --all-targets --all-features -- -D warningscargo test --locked --manifest-path harness/Cargo.toml -p harness-integration— 87 unit, 2 determinism, 1 compilation, 4 schema, and 5 supervisor tests passedcargo clippy --locked --manifest-path harness/Cargo.toml -p harness-integration --all-targets -- -D warningsmake -C harness integration-validate— 19 scenario fixtures validmake -C harness integration-test III_BIN=<pinned-engine> INTEGRATION_SCENARIO=INT-021with engine.lock revision15dc993e— passed in 2698 mscargo fmt --manifest-path llm-router/Cargo.toml --all -- --checkcargo fmt --manifest-path harness/Cargo.toml --all -- --checkactionlint 1.7.12 -colorgit diff --checkRefs MOT-4412
Summary by CodeRabbit