Field report: DeepSeek V4 Flash q2-q4 on two M1 Max 64 GB (distributed) — 427e281 results and two workarounds

[YKKW-Code]

Hi antirez,

Thanks for DwarfStar. We have been running the two-machine pipeline-parallel setup as a
nightly batch for long-document summarization since early July, and it has been reliable.

This is a field report from a machine class that is below the documented target (the README
says Metal's primary target is 96 GB or more). We upgraded from 80ebbc3 to the current
main line and hit two issues that only appear in this configuration. Both have workarounds,
so nothing here is blocking us — we are posting mainly because 64 GB machines are probably
invisible in your test matrix, and the second issue may affect future memory-budget
decisions.

Environment

• 2x MacBook Pro M1 Max, 64 GB each, macOS Tahoe 26, connected by a Thunderbolt 4 cable
• coordinator --layers 0:23 on one machine, worker --layers 24:output on the other
• Model: q2-q4-imatrix (last 6 layers Q4K), fully resident on both sides, no --ssd-streaming
• --ctx 65536, --dist-activation-bits 32, disk KV cache enabled
• Metal working-set ceiling on this machine is about 51.3 GiB; the resident model plus KV and
  buffers is 47.05 GiB, so roughly 4.3 GiB is left for prefill working memory

Results after the upgrade (both workarounds applied)

Measured against 80ebbc3 with identical scripts, same prompts, same TB link. decode is
computed as completion_tokens / (wall - prefill); for the coding tasks completion_tokens
was identical between the two builds, so those rows are a like-for-like comparison.

Workload	80ebbc3	427e281 + workarounds
Summarization prefill (17.5k / 27.4k / 15.6k tok)	148.7 / 163.0 / 147.1 t/s	155.3 / 162.7 / 153.7 t/s
Summarization decode	9.60 / 9.67 t/s	10.03 / 10.07 t/s (+4.5% / +4.1%)
Code generation decode (401-2,977 tok prompts)	10.83-12.63 t/s	11.00-12.95 t/s (+1.0-4.1%)

Output quality is unchanged; if anything the new build cites figure and page numbers slightly
more often. Thank you for the kernel work — it is a real improvement on pre-M5 hardware too.

Issue 1: distributed coordinator segfaults at startup

A coordinator started with --layers 0:23 crashes before listening:

EXC_BAD_ACCESS / SIGSEGV / KERN_INVALID_ADDRESS at 0x0000000000000020
  metal_graph_alloc_raw_cap   ds4.c:16828
  ds4_session_create          ds4.c:56585
  main                        ds4_server.c:12986

ds4.c:16828 is:

const uint64_t vocab_dim = weights->output->dim[1];

In 80ebbc3 the same line in the same function had a guard:

const uint64_t vocab_dim = weights->output ? weights->output->dim[1] : DS4_N_VOCAB;

The startup log shows why this now matters — the coordinator stopped mapping the output tensor:

80ebbc3: restricting metal model map to layers 0:23+output (52 spans, 45.72 GiB tensor span)
current: restricting metal model map to layers 0:23      (49 spans, 45.20 GiB tensor span)

So weights->output is NULL on a coordinator that does not own the output head. Single-process
runs always own it, which is presumably why this is not caught in CI. Restoring the old guard
is enough; vocab_dim only sizes output_logits_elems, which a coordinator that does not
compute logits never uses.

Issue 2: long prefill hits Metal OOM, bisected to 427e281

With the coordinator running, any prompt above roughly 1.4k-3k tokens fails:

0725 19:41:44 ds4-server: chat ctx=0..17562:17562 prompt start
ds4: Metal command batch failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

The server survives and returns HTTP 500. It appears on both the coordinator and the worker.
At around 3k tokens the failure mode is quieter and more dangerous: HTTP 200 with
gen=0 finish=stop, i.e. an empty completion rather than an error.

What we checked:

• 80ebbc3 runs the same 27,434-token prompts with zero Insufficient Memory in its logs
• Static allocation is identical between builds (planned 46.56 GiB, prefill_cap=4096,
  raw_kv_rows=4352), so this is prefill working memory, not startup residency
• DS4_METAL_TRACE_ALLOCS records nothing between prompt start and the OOM — the trace
  count is identical before and after the request. Consistent with the error being a command
  buffer callback rather than an allocation failure
• --prefill-chunk 1024 does not help (it lowers planned to 45.93 GiB and raw_kv_rows to
  1280, but the OOM is unchanged)
• DS4_METAL_EXACT_VIEW_CACHE_GIB=2 with _PROFILE=1 produces no evict lines at all, so the
  exact view cache is not involved

Bisect over 80ebbc3..0a7ad77, testing a 17,562-token prompt at each step:

Commit	Long prompt
80ebbc3	ok
e0824dd (427e281^)	ok, prefill 152.8 t/s
427e281	OOM
efdadd4, 0a7ad77	OOM

Within 427e281, disabling the contiguous copy kernels fixes it, and it is specifically the
F16 to F16 one:

Configuration	Long prompt
none disabled	OOM
DS4_METAL_DISABLE_CONTIG_F32_F16_COPY=1	OOM
DS4_METAL_DISABLE_CONTIG_F16_F16_COPY=1	ok
both disabled	ok

We now run with DS4_METAL_DISABLE_CONTIG_F16_F16_COPY=1 only. Keeping the F32 to F16 path
enabled is worth about 1.6 points of decode throughput, so the cost of the workaround is small.

Note on 64 GB machines

We understand 64 GB is below the documented target, and we are not asking for support for it.
The reason we are writing this up is that the failure is silent in a specific way: startup
succeeds, short prompts work, and only prompts past a threshold fail — with an empty
completion rather than an error in part of the range. On a machine with 30-70 GiB of headroom
the same change is invisible.

If future kernel work has a rough sense of how much prefill working memory it adds, that
number alone would let people on smaller machines predict whether an upgrade is safe.

Reproduction

# coordinator
./ds4-server --ctx 65536 -m ds4flash.gguf --role coordinator --layers 0:23 \
  --listen 10.99.0.2 9000 --host 0.0.0.0 --port 8000 --dist-activation-bits 32
# worker
./ds4 --role worker --layers 24:output --coordinator 10.99.0.2 9000 --ctx 65536 -m ds4flash.gguf
# then POST a ~17k token prompt to /v1/chat/completions

Issue 1 reproduces at coordinator startup with no request at all.

Full disclosure: the investigation above (bisect, log analysis, workaround search) was carried
out with Claude Code, and all numbers are measured on our own hardware.

Happy to answer questions or share more log detail. Longer experiment series are harder for us,
since these two machines run a scheduled workload every night, but short checks are fine.

[snowball77]

I can confirm the segfault both on metal and cuda spark

[YKKW-Code]

Thanks for confirming — good to know it is not Metal-specific. That matches the code:
weights->output is NULL on any coordinator started with --layers 0:N regardless of backend.

If you want to unblock yourself locally, restoring the guard that 80ebbc3 had is enough
(ds4.c, in metal_graph_alloc_raw_cap):

const uint64_t vocab_dim = weights->output ? weights->output->dim[1] : DS4_N_VOCAB;

Heads-up: with that patch the coordinator starts, but on our 64 GB Metal setup we then ran into
issue 2 (long prefill OOM). Whether that one also affects cuda spark, we do not know.

[michele-cei]

Confirmed on CUDA Spark / GB10 as well.

Env: 2x DGX Spark (GB10, sm_121), recent main (efdadd4 / 0a7ad77 range).
Repro:

worker: --role worker --layers 21:output --coordinator <coord_ip> 9000
coordinator: --role coordinator --layers 0:20 --listen <coord_ip> 9000 --host 0.0.0.0 --port 8000
Coordinator initializes CUDA/model mapping and then segfaults before serving.
If I move coordinator to --layers 21:output, it does not hit this specific startup crash (but that is not a valid intended routing split).

So this matches #607’s weights->output == NULL coordinator-path regression (not Metal-only).

[snowball77]

I can also confirm that the described fix works on both Metal and dgx spark. On a long context, it seems to show a different but still buggy behavior. While the coordinator prints

0726 17:09:29 ds4-server: context buffers 880.67 MiB (ctx=32768, backend=metal, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=8194)
0726 17:09:29 ds4-server: listening on http://0.0.0.0:9090
0726 17:09:41 ds4-server: chat ctx=0..13962:13962 TOOLS prompt start
0726 17:09:43 ds4-server: chat ctx=0..13962:13962 TOOLS prompt start
0726 17:09:47 ds4-server: chat ctx=0..13962:13962 TOOLS prompt start
0726 17:10:10 ds4-server: chat ctx=0..13962:13962 TOOLS prefill chunk 4096/13962 (29.3%) chunk=0.00 t/s avg=180.47 t/s 22.697s
0726 17:10:17 ds4-server: chat ctx=0..13962:13962 TOOLS prefill chunk 8192/13962 (58.7%) chunk=566.61 t/s avg=273.75 t/s 29.926s

the worker just terminates. No error message, no segfault, but it just terminates, leaving the coordinator without the later layers. The coordinator does not crash, but also cannot make any progress anymore.

[YKKW-Code]

Thanks for testing the patch on both backends.

That long-context failure might be related to issue 2 in the report. The trigger looks the same
(long prefill), although the symptom differs — on our side the worker survived and Metal returned
Insufficient Memory, whereas yours seems to disappear outright.

If it turns out to be the same underlying cause, the workaround we use may apply:

1. First, check whether the worker log has Metal command batch failed: Insufficient Memory
   before it goes silent. If the process is being killed rather than returning an error, the log
   may simply end without a message.
2. If it does look memory-related, try DS4_METAL_DISABLE_CONTIG_F16_F16_COPY=1 — that is what
   fixed it for us. Set it on both machines, not only the coordinator. Keep CONTIG_F32_F16
   enabled; disabling that one as well costs about 1.6 points of decode and was not needed.

On our setup that took a 17.5k-token prefill from a hard failure to completing at 152 t/s, with
the rest of 427e281's speedup intact.

[snowball77]

My worker is the dgx spark with cuda. There the environment flag does not apply - as far as I understand.

Coordinater is a Mac M3 Max with Metal.

[YKKW-Code]

You are right — that flag is Metal-only, so it does not apply to your CUDA worker. Our setup is
Metal on both sides, so I have no data on the CUDA path. Sorry for the noise.