Skip to content

[vscode-lm] Send attached images to the VS Code Language Model provider - #16

Merged
simurg79 merged 1 commit into
mainfrom
feat/vscode-lm-image-input
Aug 5, 2026
Merged

[vscode-lm] Send attached images to the VS Code Language Model provider#16
simurg79 merged 1 commit into
mainfrom
feat/vscode-lm-image-input

Conversation

@simurg79

@simurg79 simurg79 commented Aug 5, 2026

Copy link
Copy Markdown
Owner

What is the problem?

The VS Code Language Model provider dropped images. Any image you attached to a task was replaced with a short text placeholder describing it, so models that can actually see images never received the pixels. The provider also advertised supportsImages: false for every model regardless of what the model could do, and the webview hardcoded the same false, so the image attachment affordance stayed unavailable even for capable models.

How does this PR solve the problem?

Image blocks are now converted into a real vscode.LanguageModelDataPart.image() carrying the decoded bytes, on both the tool-result path and the ordinary message-content path. The bytes are decoded from base64 before being handed over — passing the base64 string straight through would get encoded a second time and produce a corrupt image.

Image capability is now read from the static model family table that already describes each Copilot model, in both the provider and the webview, so the two agree. A family that is not in the table falls back to false rather than claiming an unverified capability.

Anything that cannot be sent safely still degrades to the previous descriptive text placeholder instead of failing: images referenced by URL rather than embedded data, base64 that decodes to zero bytes, and media types outside the image/png, image/jpeg, image/gif, image/webp allowlist.

The one notable user-facing consequence: engines.vscode moves from ^1.84.0 to ^1.106.0. LanguageModelDataPart simply does not exist in VS Code before 1.106, so the extension cannot load against older hosts. The floor was pinned to exactly 1.106.0, found by bisecting published @types/vscode versions — the class is absent through 1.105.0 and present from 1.106.0 — rather than to 1.120.0 where the typings package currently sits, specifically to keep the install-compatibility hit as small as the feature allows. @types/vscode itself moves to ^1.120.0; typings being newer than the engine floor is normal and does not widen the runtime requirement.

How did you test the PR?

  • pnpm check-types from the repo root: 10/10 workspaces pass, on a cold cache (no cache hits).
  • Vitest across the four affected suites: 146 passed, 0 failed, 0 skippedsrc 73, packages/vscode-shim 46, webview-ui 27.
  • ESLint scoped to the changed files in all three workspaces: exit code 0, zero errors and zero warnings.

New coverage spans the conversion itself, each fallback branch, the capability derivation in the provider and the hook, and the new shim class.

Agent notes
  • packages/vscode-shim/ gains a LanguageModelDataPart class mirroring the real @types/vscode declaration (constructor(data, mimeType), statics image / text / json), wired through create-vscode-api-mock.ts, index.ts, and vscode.ts.
  • Changeset: .changeset/vscode-lm-image-input.md, minor bump for roo-cline.
  • pnpm-lock.yaml churn is from the @types/vscode bump in src/package.json and apps/vscode-e2e/package.json.
  • Engine floor 1.106.0 established empirically by bisecting @types/vscode publishes; not derived from documentation.

Convert base64 Anthropic image blocks into vscode.LanguageModelDataPart.image()
with raw decoded bytes on both the tool-result and non-tool content paths, and
derive supportsImages from the static model family table in both the provider
and the webview. Falls back to the existing descriptive text placeholder for
URL sources, malformed base64, and media types outside the png/jpeg/gif/webp
allowlist.

Raises engines.vscode to ^1.106.0 because LanguageModelDataPart does not exist
before that release.
@simurg79
simurg79 merged commit 00c649e into main Aug 5, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant