trtllm: forward routing.priority to the engine waiting queue - #28
Merged
sopwg612 merged 2 commits intoAug 3, 2026
Merged
Conversation
generate_locally read `priority` off the top level of the request, but the Rust frontend puts it at `routing.priority` (PreprocessedRequest.routing is not flattened) -- the sibling `routing.dp_rank` two lines up is read correctly. So every real request fell back to DEFAULT_REQUEST_PRIORITY and per-request priority never reached TRT-LLM, even with scheduler_config.waiting_queue_policy=priority set on the worker. The top-level key stays the health-check path, which pins 1.0. deepapi scales the [0.0, 1.0] engine priority to an integer before putting it on x-dynamo-request-priority, because nvext.agent_hints.priority is typed as an i32 while TRT-LLM rejects anything outside [0.0, 1.0]. Divide it back and clamp, so a malformed value degrades instead of erroring the request. NOT YET RUN: tensorrt_llm is not importable on the host these were written on, so the new tests need a run in the build container before merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu
sopwg612
temporarily deployed
to
external_collaborator
August 3, 2026 20:57 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Drops the DEEPINFRA_PRIORITY_SCALE divide added in the previous commit. deepapi now sends TRT-LLM's own rails (1.0 urgent, 0.0 reserved for flex, default omitted) rather than a scaled integer, because the integer-typed header cannot express the graded values the scale existed to carry. That makes this a plain pass-through with a clamp, and removes a deepinfra-local encoding this file would otherwise have to keep explaining across rebases. The clamp stays: TRT-LLM rejects anything outside [0.0, 1.0] outright, and degrading is better than failing the request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu
sopwg612
had a problem deploying
to
external_collaborator
August 3, 2026 21:06 — with
GitHub Actions
Failure
GitHub Actions
Failure
sopwg612
marked this pull request as ready for review
August 3, 2026 21:13
sopwg612
added a commit
that referenced
this pull request
Aug 4, 2026
revert: trtllm routing.priority engine hand-off (#28)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
generate_locallyreads the engine scheduling priority off the top level of the request:But the Rust frontend puts it at
routing.priority—PreprocessedRequest.routingis a nestedRoutingHints, not flattened. Its siblingrouting.dp_rankis read correctly two lines above.The result: every real inference request falls back to
DEFAULT_REQUEST_PRIORITY(0.5), so per-request priority never reaches TRT-LLM — even on a worker booted withscheduler_config.waiting_queue_policy=priority.RoutingHints.priorityis documented as "Backend engine scheduling priority forwarded to the generate call", so the intent is clear; only the read is wrong.The existing test documents the current behavior as intentional, but it is describing the bug:
The top-level key stays as the health-check path (
TrtllmHealthCheckPayloadpins 1.0) and keeps precedence.Value range
deepapi sends TRT-LLM's own scale, restricted to the rails the integer-typed header can express:
1urgent,0reserved for a future flex tier, header omitted for the 0.5 default. So this is a plain pass-through — no encoding for this file to carry across rebases.The clamp stays because TRT-LLM rejects anything outside
[0.0, 1.0]outright (tensorrt_llm/executor/request.py), and degrading beats failing the request.An earlier revision of this branch scaled the value by 10 to carry pytrtllm's graded
clamp(0.5 - 0.1 * score); that was dropped as machinery for a gradation the integer header cannot represent anyway.Testing
tensorrt_llmis not importable on the host these were written on, sotest_trtllm_handler_base.pyneeds a run in the build container before this merges. Onlypy_compilewas verified.Added:
routing.priorityunscaling (parametrized 0/5/6/10), out-of-range clamping, and health-check precedence over routing.Companion
deepinfra/backend PR — emits the header and adds the
Engine.DYNAMObranch toPriorityInfo.engine_value.🤖 Generated with Claude Code
https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu