feat(renderers): add Inkling renderer (text + image + audio)#103
Open
hallerite wants to merge 4 commits into
Open
feat(renderers): add Inkling renderer (text + image + audio)#103hallerite wants to merge 4 commits into
hallerite wants to merge 4 commits into
Conversation
Implement InklingRenderer for thinkingmachines/Inkling — a token-delimited
vision+audio model whose chat template is a flat stream of role-tagged,
<|end_message|>-terminated blocks (no BOS) with a one-off "Thinking effort
level: {N}" system line and <|content_model_end_sampling|> as the assistant
close/eos.
- Text / tools / reasoning / reasoning_effort: byte-parity with
apply_chat_template across all shapes (effort labels + raw floats, tool
cycles, multi-tool-call, content=None, tool-name resolution via
tool_call_id).
- Image + audio: byte-parity with the native InklingProcessor, including the
Qwen-VL-style placeholder expansion (image -> num_patches image pads, audio
-> one audio pad per mel frame); ships pixel_values / audio_input_ids in
MultiModalData with exact placeholder anchoring.
- Full parse_response (native-JSON tool args, types preserved) and
bridge_to_next_turn (mm carry-forward; refuses tool messages needing
cross-turn name resolution).
The InklingProcessor is native in transformers >= 5.14 (merged upstream
2026-07-15), so it loads without trust_remote_code. Bump the transformers
floor to >=5.14.0 and fast-track it past the 7-day uv exclude-newer cooldown
(drop the fast-track once 5.14.x clears the window). The full existing suite
was re-validated on 5.14.1 with no regressions.
Wiring: InklingRendererConfig (reasoning_effort str|float, validated),
MODEL_RENDERER_MAP, MULTIMODAL_MODELS={"image","audio"}, registry, __init__
exports. Tests: tests/test_inkling.py plus Inkling rows in the shared
render/config-parity/multimodal matrices (extended with an "inkling" family
and an audio modality kit). Docs updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ApprovabilityVerdict: Needs human review This PR introduces a new multimodal renderer (InklingRenderer) with ~1100 lines of new code, new configuration, parsing logic, and a significant dependency version bump. New feature additions of this scope warrant human review. You can customize Macroscope's approvability policy. Learn more. |
Condense the verbose rationale comments added with the Inkling renderer (MODEL_RENDERER_MAP / MULTIMODAL_MODELS entries, INKLING_EFFORT_MAP, the transformers pin, and the shared-test rows) to terse one-liners matching the surrounding style. Comments only — no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7883679. Configure here.
- bridge_to_next_turn copied only the outer mm_data dicts, so appending a carried-forward modality's items mutated the caller's previous_multi_modal_data lists in place. Copy the per-modality lists too. Adds a regression test (test_inkling_bridge_does_not_mutate_prior_mm_data). - _audio_hash now folds in the sampling rate, so mm_hashes["audio"] uniquely identifies the clip (matching the processor cache key); identical samples at different rates no longer collide as media IDs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # tests/test_multimodal.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Adds
InklingRendererforthinkingmachines/Inkling— a vision+audio MoE whose chat template is unlike anything else in the repo: a flat, token-delimited stream (<|message_{role}|>…<|content_{kind}|>…<|end_message|>, no BOS), a one-offThinking effort level: {N}system line emitted before the first non-system message, and<|content_model_end_sampling|>as the assistant close / eos.Full multimodal (text + image + audio), byte-parity-verified.
What it does
reasoning_effort— byte-parity withapply_chat_templateacross all shapes: effort labels and raw floats, tool declarations, multi-tool-call turns,content=None, and tool-name resolution viatool_call_id.InklingProcessor, including its Qwen-VL-style placeholder expansion (image →num_patches×<|unused_200054|>, audio → one<|unused_200053|>per mel frame). Shipspixel_values/audio_input_idsinMultiModalDatawith exact placeholder anchoring.parse_response— native-JSON tool args (types preserved, no schema coercion needed), reasoning + content channels, tool-calltoken_spans.bridge_to_next_turn— carriesMultiModalDataforward; refuses tool messages whose name would need cross-turn resolution (would diverge from a full re-render).Dependency change (please review)
The
InklingProcessor/ image / audio processors are native in transformers ≥ 5.14 (merged upstream 2026-07-15), so they load with notrust_remote_code. This bumps the sharedtransformersfloor 5.6.2 → 5.14.1 (~8 minor versions). I re-ran the entire existing suite on 5.14.1 first — no regressions to the other 20+ renderers.Because 5.14.x is inside the repo's 7-day
exclude-newercooldown,transformersis fast-tracked in[tool.uv.exclude-newer-package]souv lockcan resolve it today. This weakens the supply-chain cooldown fortransformers— there's a comment noting the fast-track should be dropped once 5.14.x clears the window. Lock churn is minimal (onlytransformers+safetensors).Tests
tests/test_inkling.py(31 tests): effort formatting, assistant structure/masks, tool-call shape, tool-name resolution, parse round-trips, bridge, config validation.conftestrender-parity barrage,test_renderer_config_parity(exercisesreasoning_effort), andtest_multimodal(newinklingfamily + audio modality kit).ruffclean.🤖 Generated with Claude Code
Note
Medium Risk
Large new renderer surface is well-tested, but raising the global
transformersminimum (~8 minor versions) is a breaking install constraint for downstream stacks pinned to older versions.Overview
Adds
InklingRendererforthinkingmachines/Inkling: token-delimited chat rendering (roles, thinking, tools,Thinking effort levelsystem line), image and audio via nativeInklingProcessorplaceholder expansion,MultiModalDatasidecars,parse_inklingfor completions, andbridge_to_next_turnwith multimodal carry-forward (returnsNonewhen tool names need prior-turn resolution).Wires
InklingRendererConfig(reasoning_effort,image_cache_max), registry/model map, and documents Inkling as multimodal in README anddocs/renderer-config.md(bridge policyall).Dependency: bumps
transformers>=5.14.0(was ≥4.50/5.6) and fast-trackstransformersin uvexclude-newer-packageuntil it clears the 7-day cooldown; lock updates includetransformers5.14.1 andsafetensors0.8.Tests: new
tests/test_inkling.py; Inkling in shared render/config parity and multimodal matrices (first audio modality kit); multimodal bridge uses renderer-specific turn-close tokens.Reviewed by Cursor Bugbot for commit f235f2a. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add Inkling renderer with text, image, and audio support
InklingRendererfor thethinkingmachines/Inklingmodel, with chat-template parity, multimodal content parts (text, image, audio), tool declaration/invocation, reasoning effort formatting, and bridge/truncation support.InklingRendererConfigin renderers/configs.py with a validatedreasoning_effortfield (named labels or float in[0.0, 0.99]) and an internalimage_cache_maxfield.parse_inklingin renderers/parsing.py to extractreasoning_content, text content, and structured tool calls from Inkling model outputs.thinkingmachines/Inklingin the model-renderer map, modality support map (image + audio), and renderer registry in renderers/base.py.transformersminimum from>=4.50.0to>=5.14.0(first release with native Inkling processor support), which is a breaking install-time requirement change.Macroscope summarized f235f2a.