[vscode-lm] Send attached images to the VS Code Language Model provider - #16
Merged
Conversation
Convert base64 Anthropic image blocks into vscode.LanguageModelDataPart.image() with raw decoded bytes on both the tool-result and non-tool content paths, and derive supportsImages from the static model family table in both the provider and the webview. Falls back to the existing descriptive text placeholder for URL sources, malformed base64, and media types outside the png/jpeg/gif/webp allowlist. Raises engines.vscode to ^1.106.0 because LanguageModelDataPart does not exist before that release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the problem?
The VS Code Language Model provider dropped images. Any image you attached to a task was replaced with a short text placeholder describing it, so models that can actually see images never received the pixels. The provider also advertised
supportsImages: falsefor every model regardless of what the model could do, and the webview hardcoded the samefalse, so the image attachment affordance stayed unavailable even for capable models.How does this PR solve the problem?
Image blocks are now converted into a real
vscode.LanguageModelDataPart.image()carrying the decoded bytes, on both the tool-result path and the ordinary message-content path. The bytes are decoded from base64 before being handed over — passing the base64 string straight through would get encoded a second time and produce a corrupt image.Image capability is now read from the static model family table that already describes each Copilot model, in both the provider and the webview, so the two agree. A family that is not in the table falls back to
falserather than claiming an unverified capability.Anything that cannot be sent safely still degrades to the previous descriptive text placeholder instead of failing: images referenced by URL rather than embedded data, base64 that decodes to zero bytes, and media types outside the
image/png,image/jpeg,image/gif,image/webpallowlist.The one notable user-facing consequence:
engines.vscodemoves from^1.84.0to^1.106.0.LanguageModelDataPartsimply does not exist in VS Code before 1.106, so the extension cannot load against older hosts. The floor was pinned to exactly 1.106.0, found by bisecting published@types/vscodeversions — the class is absent through 1.105.0 and present from 1.106.0 — rather than to 1.120.0 where the typings package currently sits, specifically to keep the install-compatibility hit as small as the feature allows.@types/vscodeitself moves to^1.120.0; typings being newer than the engine floor is normal and does not widen the runtime requirement.How did you test the PR?
pnpm check-typesfrom the repo root: 10/10 workspaces pass, on a cold cache (no cache hits).src73,packages/vscode-shim46,webview-ui27.New coverage spans the conversion itself, each fallback branch, the capability derivation in the provider and the hook, and the new shim class.
Agent notes
packages/vscode-shim/gains aLanguageModelDataPartclass mirroring the real@types/vscodedeclaration (constructor(data, mimeType), staticsimage/text/json), wired throughcreate-vscode-api-mock.ts,index.ts, andvscode.ts..changeset/vscode-lm-image-input.md,minorbump forroo-cline.pnpm-lock.yamlchurn is from the@types/vscodebump insrc/package.jsonandapps/vscode-e2e/package.json.@types/vscodepublishes; not derived from documentation.