fix(sdk): resolve custom OpenAI-compatible endpoints when model id lacks a recognized provider prefix

[asiqur-rahman]

HUMAN:

My own OpenAI-compatible endpoint kept throwing this error until I traced it to how LiteLLM resolves the provider prefix.

---

AGENT:

Why

LLMProvider.from_model() only ever calls LiteLLM's get_llm_provider() with
custom_llm_provider=None. When api_base points at a custom endpoint (a
self-hosted server such as LM Studio, or a third-party OpenAI-compatible
router) and the model id's first / segment isn't a LiteLLM-recognized
provider name (e.g. auto/coding, qwen/qwen3-coder-30b-a3b-instruct), that
call raises instead of falling back to the custom base. from_model()
swallows the exception and returns name=None, so as_litellm_call_kwargs()
omits custom_llm_provider entirely — the later, unguarded
litellm.completion() call then hits the same resolution failure and raises
to the user:

litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider
you are trying to call. You passed model=auto/coding

This reproduces with a real personal OpenAI-compatible router (Custom Model
auto/coding, a live api_base) and matches the exact symptom in the linked
issue and in prior reports (#11608, #11632, #14323 in OpenHands/OpenHands).

Summary

• LLMProvider.from_model() retries once with custom_llm_provider="openai"
  when api_base is set and the first attempt doesn't resolve a provider. A
  caller-supplied api_base already implies an OpenAI-compatible endpoint,
  and LiteLLM only strips a recognized provider prefix, so an unrecognized
  one (auto/, qwen/, etc.) reaches the endpoint unchanged.
• Without api_base, behavior is unchanged — there's no custom endpoint to
  infer, so an unresolved model id stays unresolved exactly as before.
• Added 2 regression tests to tests/sdk/llm/test_litellm_provider.py
  covering the fix and the no-api_base non-regression case.

Issue Number

Fixes #4247

How to Test

Reproduced and verified live against the real litellm package (not mocked),
via uv run python in this repo:

import litellm
litellm.get_llm_provider(
    model="auto/coding", custom_llm_provider=None,
    api_base="https://example-router.test/v1", api_key=None,
)
# -> raises litellm.BadRequestError: LLM Provider NOT provided ... model=auto/coding
# (confirmed this is the exact error from the linked issue before writing the fix)

litellm.get_llm_provider(
    model="auto/coding", custom_llm_provider="openai",
    api_base="https://example-router.test/v1", api_key=None,
)
# -> ('auto/coding', 'openai', None, 'https://example-router.test/v1')
# model id preserved verbatim, provider resolved -- this is the fix's fallback path

# Sanity checks confirming no regression for normal cases:
litellm.get_llm_provider(model="gpt-4o", custom_llm_provider=None, api_base=None, api_key=None)
# -> ('gpt-4o', 'openai', None, None)  -- unchanged
litellm.get_llm_provider(model="anthropic/claude-opus-4-5", custom_llm_provider=None, api_base=None, api_key=None)
# -> ('claude-opus-4-5', 'anthropic', None, None)  -- unchanged, prefix still stripped normally

Automated:

uv run pytest tests/sdk/llm/test_litellm_provider.py -v
# 13 passed (11 existing + 2 new), 1.19s

uv run pytest tests/sdk/llm/ -q
# 981 passed, 48 warnings (all pre-existing, unrelated to this change) in 62.32s

uv run pre-commit run --files openhands-sdk/openhands/sdk/llm/utils/litellm_provider.py tests/sdk/llm/test_litellm_provider.py
# Ruff format: Passed | Ruff lint: Passed | PEP8: Passed | pyright: Passed
# import dependency rules: Passed | Tool subclass registration: Passed

Video/Screenshots

N/A — backend-only correctness fix, no visual surface.

Type

• Bug fix
• Feature
• Refactor
• Breaking change
• Docs / chore

Notes

Credit to @moorsecopers99, whose prior PR OpenHands/OpenHands#14043 diagnosed
this exact class of bug (LM Studio namespaced model ids hitting the same
LiteLLM provider-resolution failure) and validated the same auto-prefix
approach — that PR went stale before review and targeted the pre-split
monorepo's app-server settings router, which no longer exists in this repo
layout. This reimplements the fix at the SDK's LiteLLM boundary
(LLMProvider.from_model) instead of an app-server-specific code path, per
this repo's "prefer interfaces over special cases" principle in
CONTRIBUTING.md — it applies uniformly to every consumer (CLI, agent-server,
SaaS) rather than only wherever a settings endpoint happens to intercept it.

[all-hands-bot]

🚦 CI is currently failing on this PR's latest commit.

Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request @all-hands-bot as a reviewer to have it reviewed regardless of CI status.)

This is an automated check - no AI was used to generate this comment.