fix(unsloth): detect vision and thinking for the loaded model via /api/inference/status - #36
Open
hwacookie wants to merge 1 commit into
Open
fix(unsloth): detect vision and thinking for the loaded model via /api/inference/status#36hwacookie wants to merge 1 commit into
hwacookie wants to merge 1 commit into
Conversation
hwacookie
marked this pull request as draft
August 27, 2026 20:30
…i/inference/status (Hypabolic#35, Hypabolic#37) /v1/models carries no modality or thinking metadata. An earlier revision of this PR probed /api/models/check-vision/{id} for loaded models — but for a model id that is not a resolvable HF repo (e.g. a local GGUF quant name like Qwen3.8-27B-IQ4_NL), Studio's handler falls back to fetching config.json from huggingface.co and 401s in a retry loop on EVERY listModels; with Crossbar polling every 15s (Studio has no health endpoint) that became sustained HuggingFace traffic. Both capabilities are now read from ONE local call, GET /api/inference/status, which describes the loaded backend and has no HF fallback path: - vision (Hypabolic#35): active model with is_vision: true → input ["text", "image"] - thinking (Hypabolic#37): active model with supports_reasoning: true → reasoning active_model carries the same public id as the /v1/models entry (fallback: the loaded: true entry), so the flags attach to exactly one model. Capabilities are known only for the loaded model; unloaded models pick up their real flags once loaded. Best-effort by contract: any probe failure degrades to the conservative pre-fix behaviour, never throws, never blocks registration.
hwacookie
force-pushed
the
fix/unsloth-vision-detection
branch
from
August 27, 2026 22:00
08fdf27 to
0e9cd85
Compare
hwacookie
marked this pull request as ready for review
August 27, 2026 23:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #35, fixes #37
Problem
The OpenAI-compatible
GET /v1/modelsexposes neither modality nor thinking metadata for Unsloth Studio:(image omitted: model does not support images)— VLMs appeared to "have no vision", with no error or warning anywhere.reasoning.Fix
Both capabilities are read from ONE local call that describes the loaded backend:
active_modelcarries the same public id as the/v1/modelsentry (fallback: theloaded: trueentry), so the flags attach to exactly one model.is_vision: true→ registered withinput: ["text", "image"].supports_reasoning: true→ registered withreasoning: true.Why not
/api/models/check-vision/{id}? An earlier revision of this PR probed that endpoint for loaded models. For a model id that is not a resolvable HF repo (e.g. a local GGUF quant name likeQwen3.8-27B-IQ4_NL), Studio's handler falls back to fetchingconfig.jsonfrom huggingface.co and 401s in a retry loop on EVERYlistModels— and since Crossbar polls every 15s (Studio has no health endpoint), that became sustained HuggingFace traffic./api/inference/statushas no such fallback: local, instant, zero external requests.Best-effort by contract: any probe failure (older Studio → 404, 401, refused, malformed body) degrades to the conservative pre-fix behaviour (text-only, no thinking), never throws, never blocks registration. Capabilities are known only for the loaded model; unloaded models pick up their real flags once they are loaded.
Verification
tsc --noEmitcleancheck-visionpath is NEVER requested (trap routes included), so the HF-fallback regression cannot silently returnis_visionon/api/inference/statusverified against a running Studio instance (loaded Qwen3-VL reportsis_vision: true)Changes
src/adapters/unsloth.ts— single/api/inference/statusprobe for vision + thinking; header docstests/adapters/unsloth.fixture.ts— status route withis_vision; loaded-but-not-active entrytests/adapters/unsloth.test.ts— regression tests for both capabilities, incl. "never hits check-vision" guardsCAPABILITY-MATRIX.md— Unsloth row updated