Jinja template support - #42 - #43
Open
Doelfke wants to merge 6 commits into
Open
Conversation
feat(frontend): add Jinja chat template override
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f8154eaa3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+390
to
+392
| return RenderedChat{.text = template_->render( | ||
| minja::Context::make(jinja_context(messages, options))), | ||
| .rewrite_checkpoint = std::nullopt}; |
There was a problem hiding this comment.
When a CLI or serving override does not open a <think> section—for example, an ordinary template ending with an assistant marker—enable_thinking still defaults to true, but this render result provides no indication of whether reasoning actually began; Frontend::prepare consequently sets starts_in_reasoning from the option alone. OutputSession then publishes every generated byte as reasoning until it sees </think>, leaving the normal content response empty if the model never emits that marker. Return the rendered template's actual initial channel state, or otherwise derive/restrict it for custom templates.
AGENTS.md reference: AGENTS.md:L124-L127
Useful? React with 👍 / 👎.
Comment on lines
+370
to
+372
| JinjaTemplate(std::string source, std::string source_name) | ||
| : source_name_(std::move(source_name)), | ||
| supports_reasoning_effort_(source.find("reasoning_effort") != std::string::npos) { |
There was a problem hiding this comment.
For a custom template that merely mentions reasoning_effort in a comment, string literal, or unused assignment, this substring check marks all three effort levels as supported and allows the CLI/server request through even though rendering is unchanged. Clients therefore receive a successful response for an unsupported semantic control instead of the existing reasoning_effort_not_supported behavior; capability detection needs parsed variable use or explicit template metadata rather than a raw text match.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds runtime Jinja chat-template overrides to NInfer so the CLI and server can load an external .jinja file at Engine startup (useful for Qwen3.8-27B compatibility/chatty behavior), using the vendored minja Jinja-like interpreter.
Changes:
- Vendor
third_party/minjaand integrate Jinja compilation/rendering intoqwen3_6chat template handling. - Add
--chat-template-file PATHto bothninfer(CLI) andninfer-serve, plumbed throughEngineOptionsinto target frontends. - Add tests and docs coverage for the new option and template behavior.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| third_party/minja/minja.hpp | Vendored Jinja-like template parser/render engine used for chat template overrides. |
| third_party/minja/LICENSE | MIT license for vendored minja code. |
| src/targets/qwen3_6/impl/frontend/frontend.cpp | Load chat template from filesystem path and compile as Jinja at frontend init. |
| src/targets/qwen3_6/impl/frontend/chat_template.h | Add Jinja-backed compiled template variant to CompiledChatTemplate. |
| src/targets/qwen3_6/impl/frontend/chat_template.cpp | Implement Jinja compilation/rendering and build Jinja context from chat messages/tools/options. |
| src/targets/qwen3_6/export/ninfer/targets/qwen3_6/frontend.h | Add chat_template_path to FrontendOptions. |
| include/ninfer/types.h | Add EngineOptions::chat_template_path for startup template override. |
| src/targets/qwen3_6_27b/impl/package.cpp | Forward engine chat-template path into qwen3_6 frontend options. |
| src/targets/qwen3_6_35b_a3b/impl/package.cpp | Forward engine chat-template path into qwen3_6 frontend options. |
| src/serve/serve_options.h | Store server chat-template override path in ServeOptions. |
| src/serve/serve_options.cpp | Parse --chat-template-file and add it to ninfer-serve --help text. |
| src/serve/generation_service.cpp | Plumb serve option into EngineOptions. |
| apps/cli/options.h | Store CLI chat-template override path. |
| apps/cli/options.cpp | Parse --chat-template-file and add it to ninfer --help text. |
| apps/cli/main.cpp | Plumb CLI option into EngineOptions. |
| docs/cli.md | Document CLI template override option and add it to the options table. |
| docs/serving.md | Document server template override option. |
| tests/test_serve_options.cpp | Add coverage for server flag parsing, default state, and help text. |
| tests/test_cli_options.cpp | Add coverage for CLI flag parsing, default state, and help text. |
| tests/targets/qwen3_6/test_jinja_chat_template.cpp | Validate Jinja compilation/rendering and reasoning-effort gating behavior. |
| tests/CMakeLists.txt | Register new CLI and Jinja chat template tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+385
to
+392
| bool contains(const Value & value) const { | ||
| if (is_null()) | ||
| throw std::runtime_error("Undefined value or reference"); | ||
| if (array_) { | ||
| for (const auto& item : *array_) { | ||
| if (item.to_bool() && item == value) return true; | ||
| } | ||
| return false; |
Comment on lines
+971
to
+983
| std::function<void(Value&)> visit = [&](Value& iter) { | ||
| auto filtered_items = Value::array(); | ||
| if (!iter.is_null()) { | ||
| if (!iterable_value.is_iterable()) { | ||
| throw std::runtime_error("For loop iterable must be iterable: " + iterable_value.dump()); | ||
| } | ||
| iterable_value.for_each([&](Value & item) { | ||
| destructuring_assign(var_names, context, item); | ||
| if (!condition || condition->evaluate(context).to_bool()) { | ||
| filtered_items.push_back(item); | ||
| } | ||
| }); | ||
| } |
Comment on lines
+227
to
+231
| auto i = index.get<int>(); | ||
| if (i < 0 || i >= static_cast<int>(array_->size())) | ||
| throw std::runtime_error("pop index out of range: " + index.dump()); | ||
| auto it = array_->begin() + (i < 0 ? array_->size() + i : i); | ||
| auto ret = *it; |
Comment on lines
+2778
to
+2789
| globals.set("joiner", simple_function("joiner", { "sep" }, [](const std::shared_ptr<Context> &, Value & args) { | ||
| auto sep = args.get<std::string>("sep", ""); | ||
| auto first = std::make_shared<bool>(true); | ||
| return simple_function("", {}, [sep, first](const std::shared_ptr<Context> &, const Value &) -> Value { | ||
| if (*first) { | ||
| *first = false; | ||
| return ""; | ||
| } | ||
| return sep; | ||
| }); | ||
| return Value(html_escape(args.at("text").get<std::string>())); | ||
| })); |
|
Do you think this is better than just adding a system prompt prepend option if you have something you wanted added generally and selectively fixing something like empty thinking tags injected with preserve thinking on and no reasoning content? I don't think you want anything else from the froggeric chat template as they are not relevant to ninfer or are strictly worse like telling the model to fix arguments to a call that may have correctly errored out with successful arguments. That error fix seems targeted at small qwen 3.5 models. |
Author
That may be true, but these are the standard. |
Gevil
added a commit
to Gevil/ninfer
that referenced
this pull request
Aug 24, 2026
…roued#69 Neroued#65 Neroued#57 Neroued#61) (#1) * feat(serve): report prefix-cache hits in the chat-completions usage A client cannot see the Engine's prefix cache. The OpenAI usage block carried only prompt/completion/total, so a harness computing a hit rate had nothing to read and reported a flat zero -- on a session the Engine was in fact reusing at 92%, and 98.6% over its last fifty requests. The number was there all along: the request log has printed prefix_cache_hit tokens per request from the start, and the Responses schema has reported it as input_tokens_details.cached_tokens since it was written. Emit prompt_tokens_details.cached_tokens the same way, from the same outcome.metrics.prefix_cache_hit_tokens and with the same clamp, so the two protocols agree. It is a subset of prompt_tokens, never an addend, which is what OpenAI documents and what clients subtract to get billed input. All three chat-completions call sites -- plain, tool, and the streaming usage chunk -- built the same object by hand, so they now share one helper. The Anthropic schema deliberately stays as it was: there input_tokens *excludes* cache reads, so reporting the same fact would also change what input_tokens means for every existing Messages client. Measured against the running engine: a cold 30062-token prompt reports 0, the same conversation with one turn appended reports 30060 of 30077, and the streaming path reports 30075 of 30091 -- each matching the cache= figure the engine logged for that request. * fix(serving): accept content-part arrays for tool messages The OpenAI Chat Completions contract defines tool message content as a string or an array of content parts (minItems 1) with only type "text" parts valid, and content and tool_call_id required. The OpenAI parser hard-required a string, so clients that emit tool results as content parts (Qwen Code by default, toolResultContentFormat "parts") got 400 "tool messages must contain string content" on every tool call. Route tool content through parse_content_parts() with a text-only restriction so strings and text-part arrays are accepted exactly as for user/assistant messages, empty or null content is still rejected, and non-text parts are rejected at the schema boundary with a precise 400 instead of failing later during translation. Adds regression tests for the accepted and rejected tool-content shapes and documents the tool-message content contract in docs/serving.md. * fix(serving): generalize parse_content_parts text_only into allowed_types allowlist Replace the text_only boolean restriction on parse_content_parts with a std::vector<std::string> allowed_types argument (default empty = allow all normal part types). A non-empty list rejects any part whose type is not in the list with a precise 400 that enumerates the allowed types. The tool branch now passes {"text"}, yielding the same message as before; user, assistant, and system call sites omit the argument so their behavior is unchanged. The guard runs before per-type dispatch so rejected parts are never media-acquired. * fix(serving): export parse_content_parts and unit-test the allowed_types allowlist Move parse_content_parts out of the anonymous namespace and declare it in openai_schema.h (following the existing exported parse helper parse_openai_preserve_thinking) so the allowed_types allowlist can be tested directly; behavior is unchanged. Add test_parse_content_parts_allowed_types covering the semantics the wire-level tool branch ({"text"}) cannot reach: an empty allowlist accepts every normal part type (text/image_url/video_url/input_audio), a multi-type allowlist accepts each listed type and rejects the rest with a 400 that lists the allowed types in caller order, a disallowed part is rejected at the guard before any media acquisition, and an empty array is still rejected. * feat(serve): bound each image with a Vision-token budget preprocessor_config.json ships the model's capability ceiling: size.longest_edge there is large enough that a full-resolution screen capture is never resized, so a single image occupies thousands of prompt tokens. An agent client resends every screenshot it has read on every turn, so that number decides how many turns fit the context at all, and a serving endpoint wants a policy ceiling on top of the capability one, the way every hosted API applies one: fit-and-scale rather than reject. --image-token-budget N states that policy in the unit an operator reasons about. One Vision token is a 32x32 pixel square, which validate_pixel_pipeline already pins by requiring patch_size 16 and merge_size 2, so the budget lowers image_max_pixels and nothing else: no other consumer reads that value. Zero keeps the artifact's own number, so the served behavior is unchanged unless the operator asks for a ceiling, and videos are unaffected either way. The budget is per image and deliberately does not depend on how many images a request carries. Waterfilling a shared budget across items would move the prompt prefix as a conversation grows and invalidate prefix reuse on every turn. Measured on the 5090 NVFP4 build served with --vision --max-context 32768, as usage.prompt_tokens of a one-image chat request; the same request without an image reports 53: image default budget 1280 budget 256 2880x1800 5095 1315 295 1600x1200 1955 1285 289 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(ops): fuse sigmoid gate into attention reduce epilogue * perf(ops): fuse q/k rmsnorm into one decode kernel * perf(ops): fold the moe router selection into the last D1 block The top-8 selection ran as its own single-warp grid between D1 and D3. Node-level profiling attributes 2.85 us of its 4.83 us to the node itself rather than to the selection, so the block that arrives last in D1 now runs the identical routine. The ticket uses atomicInc with wrap, so it resets itself and needs no workspace slot or host-side initialisation. * perf(ops): prefetch next projection weights from moe d4 tail * perf(ops): prefetch shared expert down weights behind moe router * build(win32): add native Windows build support Port the Windows compatibility layer from the ninfer-3090 fork onto the sm_120a (RTX 5090) codebase, keeping the original target unchanged: - CMake: MSVC preprocessor/CRT handling (NOMINMAX, WIN32_LEAN_AND_MEAN, /Zc:preprocessor, /NODEFAULTLIB:LIBCMT), static cudart on Windows, and vcpkg-based FFmpeg/libcurl discovery on WIN32 (pkg-config retained on Linux). CMAKE_CUDA_ARCHITECTURES stays pinned to 120a and the CUDA 13.1 floor is unchanged. - Add the vcpkg manifest for Windows dependency builds. - Ignore Windows development artifacts (vcpkg_installed/, models/). * fix(platform): make artifact reader, load progress, and console log portable to Windows - artifact MappedFile: Windows implementation using CreateFileW, CreateFileMappingW/MapViewOfFile for the mapped metadata, and FILE_FLAG_NO_BUFFERING overlapped ReadFile for direct 4096-byte aligned payload reads, mirroring the POSIX O_DIRECT/pread path. - load progress: detect an interactive stderr with _isatty on MSVC. - console log: use localtime_s on MSVC. No behavior change on Linux; the v2-only artifact contract is preserved. * build(win32): route library dependencies through the platform targets - Use the NINFER_CUDART_TARGET set by the top-level CMakeLists (static cudart on Windows, dynamic on Linux) in ninfer_core, ninfer_serve, and the ninfer_bench executable; keep the sm_120a NVFP4 TMA archive unconditional and pin it to the static runtime on Windows. - Link FFmpeg and libcurl through NINFER_FFMPEG_TARGET / NINFER_CURL_TARGET (vcpkg-configured on Windows, pkg-config on Linux), add ws2_32 for the media acquisition socket path on Windows, and define UTF8PROC_STATIC so the vendored utf8proc is consumed as a static library under MSVC. - request log: use _getpid on MSVC. * fix(platform): portable media acquisition and request log test - media_acquire: use winsock2/ws2tcpip headers and a one-time WSAStartup on Windows, report getaddrinfo failures numerically on MSVC (no gai_strerror), and check the media-root escape against the generic (forward-slash) path form so the containment check behaves identically on Windows and POSIX. - request log test: use _getpid on MSVC. * docs(win32): add Windows build guide and document the Windows path - Add docs/windows.md with Windows 11 x64 requirements, vcpkg setup, Visual Studio 2022 build commands, run instructions, and the Windows-specific notes (static cudart, vcpkg DLL tree, artifact I/O). - Link the guide from the documentation index. * docs(readme): document Windows 11 as a supported build platform - Requirements now list Windows 11 x64 alongside 64-bit Linux, with the MSVC/vcpkg toolchain notes; the RTX 5090 (sm_120a) target, CUDA 13.1 floor, and all model/performance content are unchanged. - Build section gains a Windows subsection (Visual Studio 2022 + vcpkg manifest) and links docs/windows.md. - Add an Upstream section noting the fork's provenance and that the 3090 fork's SM86 retargeting was intentionally not taken. * docs: point build instructions at the ninfer-windows repository The clone URLs referenced natpate/ninfer, which does not exist. Use the actual fork URL in the Linux and Windows build sections, and match the following `cd` to the resulting checkout directory name. * docs(win32): point build guide at the ninfer-windows repository The clone URL referenced natpate/ninfer, which does not exist. Use the actual fork URL and match the following `cd` to the checkout directory name. * build(win32): permit over-aligned CUDA parameters for the TMA kernels MSVC rejects formal parameters with alignment greater than 16 bytes (C2719) by default. The NVFP4/TMA kernels pass 128-byte-aligned CUtensorMap descriptor structs by value through __grid_constant__, which the Linux clang/gcc toolchains accept. Forward /d2FH4- to the MSVC host compiler for CUDA sources; every call site passes properly aligned objects, so the guarantee the diagnostic enforces holds by construction. * Revert "build(win32): permit over-aligned CUDA parameters for the TMA kernels" This reverts commit 727d6fd. * fix(ops): pass NVFP4 TMA descriptors by device pointer on MSVC MSVC cannot align a by-value alignas(128) kernel parameter (C2719 in the cudafe1 host launcher), which blocks the Windows build. The TMA unit reads the tensor map from the address named by the descriptor pointer, so on _WIN32 the launcher copies the 512-byte descriptor block to a 128-byte-aligned device buffer (cudaMallocAsync, pool-backed) and passes the kernel a pointer, freeing it stream-ordered after the launch. On other hosts the __grid_constant__ by-value parameter is unchanged; the kernel parameter spelling is centralized in the NINFER_NVFP4_TMA_DESCRIPTOR_PARAM macro so all kernel translation units share one consistent spelling, and descriptor reads go through a descriptor_block pointer alias. * fix(ops): use pair_row in the SwiGLU TMA epilogue The destination1 index was written as parent_row, an identifier from the plain linear kernel that does not exist here; it breaks compilation of the SwiGLU TMA kernel on every platform. * fix(targets): remove out-of-line qwen3_6 plan move specializations MSVC 19.44 does not emit these out-of-line `= default` explicit specializations, so the final Windows link lost the RequestBasePlan/ RequestPlan/SequencePlan move constructors (LNK2019 x6 in ninfer.exe and ninfer-serve.exe). The moves are re-defaulted in-class in runtime.h in the next commit. Destructor and bodied specializations stay as-is; MSVC emits those (registry/engine objects already resolve them). * fix(targets): default qwen3_6 plan moves in-class SequencePlan/SequencePlanner/RequestBasePlan/RequestPlan are single- unique_ptr PIMPL wrappers, so their moves and move-assignments are defaulted in-class. Every TU that moves a plan then sees the definition inline; unique_ptr moves stay well-formed while the detail impl types are incomplete. Linux codegen is unchanged (trivial unique_ptr move). * fix(targets): keep qwen3_6 plan moves declared, not defaulted, in runtime.h In-class `= default` moves pull the unique_ptr deleter into every TU that instantiates a plan: C2027 incomplete SequencePlanImpl/RequestPlanImpl in registry.cpp (std::make_unique path) and engine.cpp (std::optional's triviality traits over the per-lane plan array). Moves go back to plain declarations here; api_impl.h defines them per variant with explicit bodies in the next commit. * fix(targets): define qwen3_6 plan moves with explicit bodies MSVC 19.44 emits out-of-line explicit specializations with bodies (the plan accessors link from them) but silently drops the out-of-line `= default` versions - the original LNK2019. The bodies are exactly what `= default` would generate: a move of the unique_ptr impl member, well-formed here because the exact target TUs have the complete impl types. * serve: accept stock llama.cpp WebUI dialect (API-compatible with tools/ui) - parse chat_template_kwargs.enable_thinking (top-level or in kwargs; conflict between the two is a 400) - missing/empty 'model' defaults to the loaded artifact - max_tokens <= 0 means server default (WebUI sends -1) - /v1/models entries carry status {value: loaded} - GET /props stub (n_ctx, n_predict, speculative, modalities, chat_template enable_thinking marker) for role/thinking detection - CORS: OPTIONS handler echoes preflight Access-Control-Request-Headers (WebUI sends custom x-conversation-id), keeps Authorization/Content-Type floor Verified end-to-end against the static tools/ui build: model list, thinking toggle, non-stream + SSE streaming with reasoning_content. * serve: add --webui / --webui-dir to serve the stock llama.cpp WebUI in-process - --webui: at startup, auto-sync the prebuilt webui from the ggml-org/llama-ui Hugging Face bucket (WinHTTP, version-marker check, staged atomic swap) so the UI stays current without a rebuild; standalone mirror script: webui-update.bat - --webui-dir DIR: serve an existing built static dir - static mount at / via httplib set_mount_point; asset MIME pins - SPA fallback in the error handler (404-only, GET/HEAD, dotless, non-API paths) so it can never shadow a real asset or route; /slots, /tools and other probe paths keep honest 404s - API-key auth now gates API paths only, so the UI shell and static assets load freely (llama-server parity) - build wiring: webui_update.cpp in src/CMakeLists.txt + winhttp (MSBuild tree updated on-disk, gitignored) * fix(webui): strip scheme from host in split_url WinHttpConnect was being handed "https://huggingface.co" instead of "huggingface.co", which cannot DNS-resolve: every --webui download failed with 12005 on any machine. Affected both the initial bucket URL and CDN redirect Location headers. Caught by the portable-zip release test (first clean-dir run of the auto-download path). * docs(readme): move Upstream section up, list fork additions, add prebuilt release instructions Upstream recognition now sits right after the model table so new readers see it first. Documents what this fork adds on top of Neroued/ninfer (native Windows 11 build, runtime porting, MSVC/TMA kernel fixes, in-process llama.cpp WebUI, portable release) and adds a Prebuilt Windows release section for users who don't build. * docs: make the prebuilt-release section version-agnostic The section now says 'the latest ninfer-windows-<version>-win64-cuda131.zip' and 'the zip' instead of pinning v0.1.0, so it stays correct as releases ship. 0.2.0 (NVFP4 runtime + in-process WebUI) is published as the latest release. * docs: document per-model context ceilings in the release section The 3.8 nvfp4 and 3.6 35b models do not fit at 200k on the 32 GB 5090; only the other three can be safely raised to 200k when VRAM is free. Shipped launchers stay at 150k; this is guidance, not a release change. * feat(serve): report served context ceiling as meta.n_ctx on /v1/models objects OpenAI-dialect clients (llama-ui, hermes) probe /v1/models and /v1/models/{id} for the runtime context window and now read meta.n_ctx, mirroring the llama.cpp server model-object dialect. The value is the process --max-context ceiling, the same figure /props already reports, so clients auto-size against what the server actually serves instead of hardcoded assumptions. * fix(serve): brace ordering in models-list meta payload The context-length meta entry closed the array paren before the data entry and payload (}}}}}})}; instead of }}}}})}};), a brace/PAREN order error that a plain {}-count balance check cannot see. Verified with a stack-based nesting pass over the whole file. * docs(readme): list context window reporting in the upstream section * Update README.md * Update README.md * Update README.md * fix(build): guard winhttp webui updater to windows The auto-downloader uses WinHTTP directly, so its translation unit only compiles on Windows; non-Windows builds get an explicit runtime error for --webui instead of a compile failure. * feat(target): register qwen3.8-27b nvfp4full weights profile * feat(convert): nvfp4full encoder, calibration, converter, and verifier for qwen3.8-27b * docs(maintainer): define the qwen3.8-27b nvfp4full artifact contract * docs(maintainer): record nvfp4full gpqa-diamond 89.39% quality gate * docs(cards): add Qwen3.8-27B-nvfp4full model card * docs(cards): point nvfp4full card at its published repository * docs(cards): complete model-index source metadata * docs(cards): point nvfp4full card at its published fork branch * docs(readme): list the published qwen3.8 nvfp4full artifact * docs: compare gpqa against the official's republished 90.40% score * fix(serve): type tool-call parameters by their declared schema Consult each tool's JSON Schema to decide how a generated parameter value is typed, instead of type-sniffing every value: - Only parameters whose declared "type" is a valid non-string JSON Schema type (integer, number, boolean, array, object, null) are deserialized; string-typed, unknown/invalid, and absent-typed params preserve the model's raw text so the string contract reaches the client intact. Full JSON Schema validation remains the client's job. - Boolean-declared params coerce Python-style scalars the model emits (True/False, 1/0) to JSON booleans (vLLM qwen3coder coercion); the literal null is JSON null (a valid value for a nullable boolean); any other value stays raw text. All comparisons are case-insensitive (True/TRUE, Null/NULL). - The per-parameter type map stores the full declared type set, so nullable types like ["boolean","null"] are handled independently of the type-array order. Review findings applied: - build_tool_param_type_map: whitelist valid non-string types (rejects misspelled/invalid types like "strnig" that the old != "string" test admitted) and support "type" as a string or an array of strings. - build_tool_param_type_map: replace a tool's entry when its name is seen again, so a redefinition cannot leak stale non-string permissions. - Reject duplicate tool names in openai_schema.cpp and anthropic_schema.cpp (responses_schema.cpp already did), matching the Responses contract. - Schema-driven tests exercise build_tool_param_type_map with real ToolDefinition schemas; a render regression test proves a string-typed taskId="1" survives make_messages_response as a JSON string. - Document the narrower parsing contract in docs/serving.md. * feat(frontend): add Jinja chat template override Vendors third_party/minja (3 077 lines) and adds the --chat-template-file option so ninfer-serve/ninfer can replace the artifact's embedded prompt renderer with a self-contained Jinja chat template (upstream PR Neroued#43). Adds the Jinja frontend to the qwen3_6 target, plumbs chat_template_path through the CLI/serve options, and documents the override in cli/serving. * fix(templates): honor reasoning effort in custom Jinja chat templates Probe the template source for a reasoning_effort reference and expose the supported levels (low/medium/xhigh, default medium) through PromptCapabilities, rendering the effort into the Jinja context when supported. Reject the effort only when the template does not reference reasoning_effort. * docs: trim verbose custom-chat-template sections Remove the early Minja feature-list and template-contract paragraphs from docs/cli.md and docs/serving.md, and drop --chat-template-file from the serving example. * feat(serve): advertise max_model_len in the /v1/models contract Report the configured --max-context ceiling as max_model_len on /v1/models and /v1/models/{id} so clients can size requests against the per-sequence cap. Documented by 'docs(serving): document max_model_len in models contract'. * fix(templates): distinguish break from continue in minja loop-control tokens The LoopControlTemplateToken constructor hardcoded Type::Break for both control types, so {% continue %} tokens misreported their token type. Derive the token type from the control type. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * docs(serving): document max_model_len in models contract Advertised schema change (max_model_len on /v1/models and /v1/models/{id}) was missing from the serving documentation, which AGENTS.md requires be updated together with the schema. Adds a Models subsection defining the model object fields and that max_model_len reflects the configured --max-context per-sequence ceiling. * fix(templates): compare minja JSON arrays/objects with native equality Value::operator== used a hand-rolled element loop gated on to_bool(), so non-truthy values (0, false, "") always compared as unequal even when identical. Delegate to the native JSON array/object == operators for element-wise equality. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * build: pin docker.io base images for podman registry resolution Carried over from 272ca7b. The sws_scale/JPEG fix in that commit was superseded by upstream b2b96ba (align swscale destination buffers), so only the Dockerfile registry pin is re-applied here. * fix(frontend): derive starts_in_reasoning from the rendered prompt P1 follow-up to the local Jinja chat template work (PR Neroued#43 merge, 96d4f30): seed starts_in_reasoning from the actually-rendered prompt (open/close reasoning markers across all template variants) instead of render options alone, on both the text and media paths. Tree matches running image ninfer:latest (3cd13cb3, built 2026-08-18). * docs: personal-fork adoption record (ADOPTION.md) + README note Records what this fork is (personal fork of Neroued/ninfer merging multiple open PRs from different community forks, cherry-picking improvements and features for the Qwen3.8-27B-nvfp4full lane), the adoption pipeline (one tagged merge commit per PR, merge-tree conflict policy, per-batch lane build + verification), and the tier plan (Tier 1 batch in progress). Linked from a note at the top of README.md. * test(serve): cover usage.prompt_tokens_details.cached_tokens; fix three stale test bugs PR#55 shipped the cached_tokens observability with no test coverage. New test_usage_cached_tokens_details pins the contract: the field is emitted in the non-streaming response and the streaming usage chunk, is a subset of prompt_tokens (clamped into [0, prompt_tokens]), and the Anthropic schema deliberately omits it (input_tokens there excludes cache reads). Also fixes three pre-existing test bugs in the same file (the file never compiled/ran cleanly upstream): - unqualified SpeculativeBackend at file scope -> ninfer::SpeculativeBackend - the reasoning_effort/enable_thinking conflict now surfaces at parse time (same conflicting_template_option code); the test still expected the resolve-time path and crashed the suite - props n_ctx lives in default_generation_settings (matching the llama.cpp /props schema), not inside params * docs(adoption): Tier 1 done - verification summary + PR link * fix(frontend): strip stray vision-pad markers instead of rejecting the request --------- Co-authored-by: Aleksandr Iakimov <sociopacific@gmail.com> Co-authored-by: jpf <jf@incyan.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: MichaelDementii <m.dementii@arvilab.com> Co-authored-by: MichaelDementii <136074657+MichaelDementii@users.noreply.github.com> Co-authored-by: natpate <80576637+natpate@users.noreply.github.com> Co-authored-by: Hyeseong Kim <hey@hyeseong.kim> Co-authored-by: Gideon Zenz <gzenz@gmx.de> Co-authored-by: David Oelfke <doelfke@gmail.com> Co-authored-by: David Oelfke <Doelfke@users.noreply.github.com> Co-authored-by: SolettaSolaris <17378474+SolettaSolaris@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Qwen3.8-27b is amazing, but it's very chatty and still has compatibility issues. I have successfully tested these changes with https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates, a fork of the popular froggeric Qwen template fix.