Skip to content

fix(unsloth): detect vision and thinking for the loaded model via /api/inference/status - #36

Open
hwacookie wants to merge 1 commit into
Hypabolic:mainfrom
hwacookie:fix/unsloth-vision-detection
Open

fix(unsloth): detect vision and thinking for the loaded model via /api/inference/status#36
hwacookie wants to merge 1 commit into
Hypabolic:mainfrom
hwacookie:fix/unsloth-vision-detection

Conversation

@hwacookie

@hwacookie hwacookie commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Fixes #35, fixes #37

Problem

The OpenAI-compatible GET /v1/models exposes neither modality nor thinking metadata for Unsloth Studio:

Fix

Both capabilities are read from ONE local call that describes the loaded backend:

GET /api/inference/status  →  { active_model, is_vision, supports_reasoning, ... }

Why not /api/models/check-vision/{id}? An earlier revision of this PR probed that endpoint for loaded models. For a model id that is not a resolvable HF repo (e.g. a local GGUF quant name like Qwen3.8-27B-IQ4_NL), Studio's handler falls back to fetching config.json from huggingface.co and 401s in a retry loop on EVERY listModels — and since Crossbar polls every 15s (Studio has no health endpoint), that became sustained HuggingFace traffic. /api/inference/status has no such fallback: local, instant, zero external requests.

Best-effort by contract: any probe failure (older Studio → 404, 401, refused, malformed body) degrades to the conservative pre-fix behaviour (text-only, no thinking), never throws, never blocks registration. Capabilities are known only for the loaded model; unloaded models pick up their real flags once they are loaded.

Verification

  • Full suite: 711 passed / 8 skipped (pre-existing live-server skips); tsc --noEmit clean
  • Regression tests assert the check-vision path is NEVER requested (trap routes included), so the HF-fallback regression cannot silently return
  • is_vision on /api/inference/status verified against a running Studio instance (loaded Qwen3-VL reports is_vision: true)

Changes

  • src/adapters/unsloth.ts — single /api/inference/status probe for vision + thinking; header docs
  • tests/adapters/unsloth.fixture.ts — status route with is_vision; loaded-but-not-active entry
  • tests/adapters/unsloth.test.ts — regression tests for both capabilities, incl. "never hits check-vision" guards
  • CAPABILITY-MATRIX.md — Unsloth row updated

…i/inference/status (Hypabolic#35, Hypabolic#37)

/v1/models carries no modality or thinking metadata. An earlier revision of
this PR probed /api/models/check-vision/{id} for loaded models — but for a
model id that is not a resolvable HF repo (e.g. a local GGUF quant name like
Qwen3.8-27B-IQ4_NL), Studio's handler falls back to fetching config.json from
huggingface.co and 401s in a retry loop on EVERY listModels; with Crossbar
polling every 15s (Studio has no health endpoint) that became sustained
HuggingFace traffic.

Both capabilities are now read from ONE local call, GET /api/inference/status,
which describes the loaded backend and has no HF fallback path:

- vision (Hypabolic#35): active model with is_vision: true → input ["text", "image"]
- thinking (Hypabolic#37): active model with supports_reasoning: true → reasoning

active_model carries the same public id as the /v1/models entry (fallback:
the loaded: true entry), so the flags attach to exactly one model. Capabilities
are known only for the loaded model; unloaded models pick up their real flags
once loaded. Best-effort by contract: any probe failure degrades to the
conservative pre-fix behaviour, never throws, never blocks registration.
@hwacookie
hwacookie force-pushed the fix/unsloth-vision-detection branch from 08fdf27 to 0e9cd85 Compare August 27, 2026 22:00
@hwacookie hwacookie changed the title fix(unsloth): detect vision models via /api/models/check-vision fix(unsloth): detect vision and thinking for the loaded model via /api/inference/status Aug 27, 2026
@hwacookie
hwacookie marked this pull request as ready for review August 27, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant