Skip to content

feat(renderers): add Inkling renderer (text + image + audio)#103

Open
hallerite wants to merge 4 commits into
mainfrom
feat/inkling-renderer
Open

feat(renderers): add Inkling renderer (text + image + audio)#103
hallerite wants to merge 4 commits into
mainfrom
feat/inkling-renderer

Conversation

@hallerite

@hallerite hallerite commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Adds InklingRenderer for thinkingmachines/Inkling — a vision+audio MoE whose chat template is unlike anything else in the repo: a flat, token-delimited stream (<|message_{role}|>…<|content_{kind}|>…<|end_message|>, no BOS), a one-off Thinking effort level: {N} system line emitted before the first non-system message, and <|content_model_end_sampling|> as the assistant close / eos.

Full multimodal (text + image + audio), byte-parity-verified.

What it does

  • Text / tools / reasoning / reasoning_effort — byte-parity with apply_chat_template across all shapes: effort labels and raw floats, tool declarations, multi-tool-call turns, content=None, and tool-name resolution via tool_call_id.
  • Image + audio — byte-parity with the native InklingProcessor, including its Qwen-VL-style placeholder expansion (image → num_patches × <|unused_200054|>, audio → one <|unused_200053|> per mel frame). Ships pixel_values / audio_input_ids in MultiModalData with exact placeholder anchoring.
  • parse_response — native-JSON tool args (types preserved, no schema coercion needed), reasoning + content channels, tool-call token_spans.
  • bridge_to_next_turn — carries MultiModalData forward; refuses tool messages whose name would need cross-turn resolution (would diverge from a full re-render).

Dependency change (please review)

The InklingProcessor / image / audio processors are native in transformers ≥ 5.14 (merged upstream 2026-07-15), so they load with no trust_remote_code. This bumps the shared transformers floor 5.6.2 → 5.14.1 (~8 minor versions). I re-ran the entire existing suite on 5.14.1 first — no regressions to the other 20+ renderers.

Because 5.14.x is inside the repo's 7-day exclude-newer cooldown, transformers is fast-tracked in [tool.uv.exclude-newer-package] so uv lock can resolve it today. This weakens the supply-chain cooldown for transformers — there's a comment noting the fast-track should be dropped once 5.14.x clears the window. Lock churn is minimal (only transformers + safetensors).

Tests

  • New tests/test_inkling.py (31 tests): effort formatting, assistant structure/masks, tool-call shape, tool-name resolution, parse round-trips, bridge, config validation.
  • Inkling added to the shared conftest render-parity barrage, test_renderer_config_parity (exercises reasoning_effort), and test_multimodal (new inkling family + audio modality kit).
  • Full suite: 2710 passed, 114 skipped, 1 xfailed (2584 pre-Inkling). Multimodal parity across all families: 91 passed. ruff clean.

🤖 Generated with Claude Code


Note

Medium Risk
Large new renderer surface is well-tested, but raising the global transformers minimum (~8 minor versions) is a breaking install constraint for downstream stacks pinned to older versions.

Overview
Adds InklingRenderer for thinkingmachines/Inkling: token-delimited chat rendering (roles, thinking, tools, Thinking effort level system line), image and audio via native InklingProcessor placeholder expansion, MultiModalData sidecars, parse_inkling for completions, and bridge_to_next_turn with multimodal carry-forward (returns None when tool names need prior-turn resolution).

Wires InklingRendererConfig (reasoning_effort, image_cache_max), registry/model map, and documents Inkling as multimodal in README and docs/renderer-config.md (bridge policy all).

Dependency: bumps transformers>=5.14.0 (was ≥4.50/5.6) and fast-tracks transformers in uv exclude-newer-package until it clears the 7-day cooldown; lock updates include transformers 5.14.1 and safetensors 0.8.

Tests: new tests/test_inkling.py; Inkling in shared render/config parity and multimodal matrices (first audio modality kit); multimodal bridge uses renderer-specific turn-close tokens.

Reviewed by Cursor Bugbot for commit f235f2a. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add Inkling renderer with text, image, and audio support

  • Adds renderers/inkling.py implementing InklingRenderer for the thinkingmachines/Inkling model, with chat-template parity, multimodal content parts (text, image, audio), tool declaration/invocation, reasoning effort formatting, and bridge/truncation support.
  • Adds InklingRendererConfig in renderers/configs.py with a validated reasoning_effort field (named labels or float in [0.0, 0.99]) and an internal image_cache_max field.
  • Adds parse_inkling in renderers/parsing.py to extract reasoning_content, text content, and structured tool calls from Inkling model outputs.
  • Registers thinkingmachines/Inkling in the model-renderer map, modality support map (image + audio), and renderer registry in renderers/base.py.
  • Risk: bumps the transformers minimum from >=4.50.0 to >=5.14.0 (first release with native Inkling processor support), which is a breaking install-time requirement change.

Macroscope summarized f235f2a.

Implement InklingRenderer for thinkingmachines/Inkling — a token-delimited
vision+audio model whose chat template is a flat stream of role-tagged,
<|end_message|>-terminated blocks (no BOS) with a one-off "Thinking effort
level: {N}" system line and <|content_model_end_sampling|> as the assistant
close/eos.

- Text / tools / reasoning / reasoning_effort: byte-parity with
  apply_chat_template across all shapes (effort labels + raw floats, tool
  cycles, multi-tool-call, content=None, tool-name resolution via
  tool_call_id).
- Image + audio: byte-parity with the native InklingProcessor, including the
  Qwen-VL-style placeholder expansion (image -> num_patches image pads, audio
  -> one audio pad per mel frame); ships pixel_values / audio_input_ids in
  MultiModalData with exact placeholder anchoring.
- Full parse_response (native-JSON tool args, types preserved) and
  bridge_to_next_turn (mm carry-forward; refuses tool messages needing
  cross-turn name resolution).

The InklingProcessor is native in transformers >= 5.14 (merged upstream
2026-07-15), so it loads without trust_remote_code. Bump the transformers
floor to >=5.14.0 and fast-track it past the 7-day uv exclude-newer cooldown
(drop the fast-track once 5.14.x clears the window). The full existing suite
was re-validated on 5.14.1 with no regressions.

Wiring: InklingRendererConfig (reasoning_effort str|float, validated),
MODEL_RENDERER_MAP, MULTIMODAL_MODELS={"image","audio"}, registry, __init__
exports. Tests: tests/test_inkling.py plus Inkling rows in the shared
render/config-parity/multimodal matrices (extended with an "inkling" family
and an audio modality kit). Docs updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@macroscopeapp

macroscopeapp Bot commented Jul 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

This PR introduces a new multimodal renderer (InklingRenderer) with ~1100 lines of new code, new configuration, parsing logic, and a significant dependency version bump. New feature additions of this scope warrant human review.

You can customize Macroscope's approvability policy. Learn more.

Condense the verbose rationale comments added with the Inkling renderer
(MODEL_RENDERER_MAP / MULTIMODAL_MODELS entries, INKLING_EFFORT_MAP, the
transformers pin, and the shared-test rows) to terse one-liners matching
the surrounding style. Comments only — no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7883679. Configure here.

Comment thread renderers/inkling.py
Comment thread renderers/inkling.py
- bridge_to_next_turn copied only the outer mm_data dicts, so appending a
  carried-forward modality's items mutated the caller's
  previous_multi_modal_data lists in place. Copy the per-modality lists too.
  Adds a regression test (test_inkling_bridge_does_not_mutate_prior_mm_data).
- _audio_hash now folds in the sampling rate, so mm_hashes["audio"] uniquely
  identifies the clip (matching the processor cache key); identical samples
  at different rates no longer collide as media IDs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hallerite
hallerite requested review from eligotts and snimu July 22, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant