opencode2-vision-fallback

An OpenCode 2.0 (preview / next) native plugin that gives text-only models image understanding.

Paste an image while chatting with a text-only model (DeepSeek, GLM, …) and the plugin transparently transcribes it through a vision-capable model before the chat model sees the request. The chat model receives plain text — no tool calls, no manual file paths, no "I can't view images" refusals.

you paste an image with DeepSeek active
        │
        ▼
vision-fallback (session context hook)
        │  • active model capabilities lack "image"? → transcribe
        │  • sends image to YOUR vision model (OpenAI-compatible call)
        │  • replaces the image part IN PLACE with the description
        ▼
DeepSeek receives: "[Vision model description of attached image:]
                    A screenshot of a settings dialog showing…"

Why not the existing vision plugins?

OpenCode 2.0's plugin loader requires the new { id, setup(ctx) } object API and rejects v1-style plugins (function exports) with SchemaError: Expected object, got async function. The existing vision plugins (opencode-see-image, opencode-eyesight, opencode-easy-vision, …) are all v1-style and cannot load on opencode2. This plugin is written natively against the opencode2 API.

Requirements

• opencode2 (@opencode-ai/cli, 2.0 preview / next channel)
• A vision-capable model served through any OpenAI-compatible provider configured in your opencode.json (OmniRoute, LiteLLM, OpenAI, Ollama, etc.)

Install

mkdir -p ~/.config/opencode/plugins
curl -o ~/.config/opencode/plugins/vision-fallback.js \
  https://raw.githubusercontent.com/luishmcmoreno/opencode2-vision-fallback/main/vision-fallback.js

Add it to the plugins array in ~/.config/opencode/opencode.json:

{
  "plugins": ["~/.config/opencode/plugins/vision-fallback.js"]
}

Point it at your vision model — either edit DEFAULT_MODEL at the top of the file, or set an env var:

export VISION_FALLBACK_MODEL="omniroute/antigravity/gemini-3.6-flash-high"

The background service watches plugin files and hot-reloads — no restart needed. Verify with:

opencode2 plugin list | grep vision-fallback

Supported media

The plugin transcribes any image/*, video/*, audio/* and application/pdf media part that appears in a request. What that means in practice today (verified on opencode2 next-17055):

Media	Behavior with a text-only chat model
Images (image/*)	✅ Transcribed by this plugin — the gap opencode2 has, and the reason it exists
PDF	✅ Already handled natively by opencode2 (text extraction); the plugin also covers PDFs that arrive as media parts (e.g. TUI paste)
Audio	✅ Already handled natively by opencode2 (transcription); the plugin covers audio media parts as a fallback
Video	⚠️ opencode2's attachment system is currently image-only, so video never reaches messages. The plugin's video transcription is implemented and verified against Gemini directly — it starts working automatically once opencode2 accepts video attachments

Media routing formats (verified against Gemini through an OpenAI-compatible gateway):

• image/*, video/*, application/pdf → image_url content part with a typed data URL
• audio/* → input_audio content part

How it decides when to transcribe

• The session.context hook exposes the active model and mutable messages.

• The plugin reads model capabilities from opencode2's catalog (capabilities.input).

• "image" present → the request passes through untouched (native vision path).

• Only "text" (or unknown) → every image part ({type: "media", mediaType: "image/*", data: <base64>}) is replaced in place with:

  [Vision model description of attached image "screenshot.png":]
  …description…
  [/Vision model description]
  

• Descriptions are cached by content hash for the process lifetime — follow-up turns re-use the cache, no extra API calls.

• Failures never block the chat: the image is replaced with a short error note instead.

Tips

• Keep native vision for capable models. For custom OpenAI-compatible providers, declare modalities in opencode.json so opencode2 knows the model takes images:

  "models": {
    "my-gemini": {
      "name": "my-gemini",
      "modalities": { "input": ["text", "image"], "output": ["text"] }
    }
  }

  Otherwise the plugin will (harmlessly, but wastefully) transcribe images for that model instead of letting it see pixels natively.

• Debug log: tail -f /tmp/vision-fallback.log — look for "event":"transcribe" when the plugin fires.

Tested with

• Chat model: opencode/deepseek-v4-flash-free (text-only)
• Vision backend: Gemini via OmniRoute (@ai-sdk/openai-compatible provider)
• opencode2 v0.0.0-next-17055

License

MIT