Add queue-mode serverless scaling support to SDK and CLI - #495
Draft
LucasArmandVast wants to merge 5 commits into
Draft
Add queue-mode serverless scaling support to SDK and CLI#495LucasArmandVast wants to merge 5 commits into
LucasArmandVast wants to merge 5 commits into
Conversation
LucasArmandVast
commented
Aug 24, 2026
LucasArmandVast
commented
Contributor
- RouteResponse: surface autoscaler queue-mode responses as QUEUED with queue_position/queue_depth (READY still keyed on url presence)
- Endpoint.submit() / Endpoint.task_status(): explicit submit-then-poll helpers over the existing /route/ protocol
- vastai create/update endpoint: --scaling_mode, --scaler, --requests_per_worker, --queue_delay_target, --idle_timeout, --cold_ttl build the endpoint scaling_config
- tests for QUEUED parsing and submit/task_status delegation
- RouteResponse: surface autoscaler queue-mode responses as QUEUED with queue_position/queue_depth (READY still keyed on url presence) - Endpoint.submit() / Endpoint.task_status(): explicit submit-then-poll helpers over the existing /route/ protocol - vastai create/update endpoint: --scaling_mode, --scaler, --requests_per_worker, --queue_delay_target, --idle_timeout, --cold_ttl build the endpoint scaling_config - tests for QUEUED parsing and submit/task_status delegation
…ast.py changes - vastai/api/endpoints.py: create_endpoint/update_endpoint accept a scaling_config dict passed through to /endptjobs/ - vastai/cli/commands/endpoints.py: --scaling_mode/--scaler/ --requests_per_worker/--queue_delay_target/--idle_timeout/--cold_ttl on create/update endpoint, built into scaling_config - vast.py restored to master (deprecated shim, no new features) - CLI blob tests for scaling_config construction
Long-running worker requests are silent for minutes while the model computes; NAT/conntrack idle timeouts on the path reap those connections, the pyworker sees a client disconnect and cancels the task, and the SDK re-routes and restarts the work in a loop. Observed live with 300s tasks dying every ~80-90s. socket_factory sets SO_KEEPALIVE with 30s idle / 15s interval / 8 probes so the kernel keeps the path warm; degrades with a warning on aiohttp versions without socket_factory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.