deepseek_v4: wire tool calling via checkpoint-native DSML encoding

[gcaponi]

What

Tool calling for the DeepSeek V4 engine (/v1/chat/completions with tools →
tool_calls), closing the "tools and grammar are rejected" serving gap for V4.

Instead of inventing a prompt format, this vendors the checkpoint's own DSML
encoding (encoding/encoding_dsv4.py inside deepseek-ai/DeepSeek-V4-Flash-0731)
into c/v4_dsml.py:

• tool schemas render into the system turn with the official ## Tools DSML template
• assistant tool_calls history renders as <｜DSML｜invoke> blocks
• tool role results merge into user turns as <tool_result> blocks
  (V4 has no standalone tool role — this is the native merge)
• reply parsing splits the \n\n<｜DSML｜tool_calls> block and decodes
  string="true|false" parameters back to JSON arguments

Changes

• c/v4_dsml.py (new): vendored reference encoding. Encode side kept byte-exact
  (verified against the checkpoint's own fixtures); parse side made fault-tolerant
  for gateway use — malformed/truncated DSML degrades to visible content + stderr
  log, never raises to the client (same philosophy as GLM salvage-off).
• c/openai_server.py (+~90 lines):
  • render_chat_v4_tools(): used when tools are declared or the history contains
    tool/tool_calls messages; plain chat rendering is byte-identical to before
  • parse_tool_calls_any(): routes reply parsing per arch (GLM unchanged)
  • streaming tool-marker suppression now uses the per-arch opener
    (<｜DSML｜tool_calls> for V4)
  • tool_choice: auto/none supported; required/forced returns an honest 400
• c/tests/test_openai_tools_v4_e2e.py (new): mirrors test_openai_tools_e2e.py
  against the deepseek_v4 arch with a DSML-speaking mock engine (4 tests).

The Anthropic /v1/messages endpoint gets tool_use blocks for free, since it
translates onto the same OpenAI path.

Validation

• test_openai_tools_v4_e2e: 4/4 pass (non-stream call, <tool_result> round
  trip, streamed marker suppression across 12 chunks, plain chat unchanged)
• test_openai_tools_e2e (GLM regression): 6/6 pass
• test_openai_server: 122/123 — the single error (resource_plan.physical_cpu_count
  mock) reproduces identically on unpatched main
• Real checkpoint probe: tools/try_tool_calling.py PASS against
  DeepSeek-V4-Flash-0731 (284B) served CPU-only on consumer hardware (6-core Ryzen 5,
  32 GB RAM, --ram 22, greedy): turn 1 get_weather({"city": "Rome"}), turn 2
  answer consumes the tool result ("sunny, 31°C")

Scope / follow-ups

• tool_choice: required / forced function: 400 for now, prompt-level support is a
  small follow-up
• grammar/response_format on V4 remains rejected (unchanged)
• serving stays greedy + 1 KV slot (unchanged)

Nota strategica

Questa PR è la SPEC-41 (prima PR mergiata) e chiude il cerchio con il commento su #175:
arriviamo al progetto con un dataset pubblico E con codice funzionante misurato sul
modello reale. Se il maintainer chiede modifiche, si itera sul branch — è il gioco.

[dcutugno]

This one seems better than my previous #948, so better drop mine or merge this...

[gcaponi]

Thanks @dcutugno — and congratulations on #948, we independently converged on the exact same design (vendored official DSML encoding, tolerant parse, per-arch dispatch). No point duplicating: I'm happy to close this in favor of the merged work.

What this PR still adds on top: a standalone e2e suite (4 tests, DSML-speaking mock engine over the SERVE wire protocol, incl. streamed marker suppression across chunks) and a live validation of the full two-turn tool loop on the real 284B checkpoint served CPU-only on consumer hardware (6-core, 32 GB RAM, greedy) — try_tool_calling.py
PASS. If useful, I'll rebase the e2e suite onto dev and open a tests-only follow-up PR, plus file the CPU-serving numbers where they belong.

[dcutugno]

What i recommend is keep my 948, but refactor it with your vendor encoder, i'm currently doing it, stay tuned!

[dcutugno]

Done this PR can be closed!