Field report: Qwen3.8-2.4T-A95B in TP4 on four M3 Ultras (early numbers)

[guruswami-ai]

Qwen3.8-2.4T-A95B in TP4 on four M3 Ultras

Early field report, 13 August 2026. One certified JACCL boot. A 64k/128k
prefill ladder is still running as this is written. An 8-bit lm_head +
embed_tokens patch is staged from official BF16 and not yet loaded. We will
follow up with those numbers rather than wait to post.

Stack: mlx 0.32.0, mlx-lm 0.31.3, macOS 26.6.1 (25G76), kernel 25.6.0,
4× Mac Studio M3 Ultra 512 GB unified memory, Thunderbolt 5 full mesh, JACCL/RDMA,
MLX_METAL_FAST_SYNCH=0.

This is a field report, not a recommendation to do the same.

We wanted to know whether the open-weight Qwen3.8 2.4T model can run at all on
the largest consumer Apple silicon we can cable together. It can. What follows is
how we built the checkpoint, what we measured, and an honest reading of what that
is worth. We do not have perplexity, MMLU, or any other capability eval:
only stability, throughput, and a handful of behavioural probes.

The open checkpoint is not Qwen3.8-Max. Max is the hosted product (vision,
optional non-thinking, 1M context, official tools). Qwen/Qwen3.8-2.4T-A95B is
text-only and thinking-only: the chat template errors if you pass
enable_thinking=false, and every generation prompt opens <think>.

---

The lesson, first

Hosting a 2.4T model on four M3 Ultras is possible. Using it as Qwen3.8 is a
different claim, and we did not establish it.

On this hardware the body must be about 4-bit. That is arithmetic, not
taste:

Precision	Weights (order of)	Four × ~512 GB unified
BF16	~4.8 TB	no
FP8	~2.4 TB	no (weights alone, before KV/OS)
Affine 4-bit	~1.2 TB	yes, ~342 GB resident / rank
8-bit body	~2.4 TB	no (~600 GB / rank after TP4 shard)

If 4-bit wrecks the job this model exists for (long think, sampled decode),
then this class of weight does not have a useful home on four Studios. That is
the consumer-hardware ceiling, not a convert bug we can wish away. An 8-bit
body does not fit. An 8-bit head on a 4-bit body does; that swap is
staged and not yet measured.

We still think there may be a narrow air-gapped use (below). We do not think
this is how anyone who can use a data centre or Qwen Cloud should run 2.4T.

---

How we built it

Stock mlx_lm.convert cannot ingest this checkpoint on a 512 GB node: Model()
allocates the dense tree, and Hugging Face quant_method=fp8 (block-128) is not
handled. We streamed official Qwen/Qwen3.8-2.4T-A95B-FP8:

1. Dequantize experts with mx.from_fp8 × weight_scale_inv (128×128 blocks).
2. Stack routed experts into SwitchGLU [512, out, in].
3. Affine-quantize, group 64, 4-bit; write 5 GB shards; drop mtp.*.
4. Set model_type=qwen3_5_moe so mlx-lm 0.31.3's qwen3_5_moe.py loads it
   (MODEL_REMAPPING has no qwen3_5_moe_text).

Convert wall time 803 s. Output: 277 shards, 1.20 TiB, 2.420×10¹²
parameters. Per-tensor 8-bit was applied only to MoE router gates
(mlp.gate, shared_expert_gate), copying qwen3_5.quant_predicate.
lm_head and embed_tokens are 4-bit (vocab 248,320). mlx-lm's mixed
quant recipes keep lm_head at higher bits; this convert did not.

mlx_lm.share broadcast the replica over JACCL in 273 s (~4.5 GB/s). RDMA
protection domains are not reclaimed on process exit; after a multi-hundred-GB
share we rebooted the fleet before TP4. mlx.distributed_config is once per
boot. We did not bump mlx / mlx-lm.

Architecture as loaded: 92 layers, hybrid 23 full attention / 69 Gated
DeltaNet, GQA 64 Q / 4 KV (head_dim 256), 512 experts top-10 + 1 shared,
hidden 8192. Native context 262,144 (1,010,000 is an extension, untested).
Not MLA. GDN ArraysCache is not trimmable (can_trim_prompt_cache is false)
but is extendable.

---

What we measured

One certified JACCL boot. prefill_step_size=512 (mlx-lm already chunks the
prompt; the CLI on 0.31.3 does not expose the flag: pass it in Python).
Official card sampler is temperature=1.0, top_p=0.95, top_k=20.
reasoning_effort=low unless noted. All quality probes below are greedy
except the sampler sweep.

Load and fabric

all_sum of ones	4.0 on every rank, 0.75 s
Load	190–192 s, 342.4 GB active / 366 GB peak, all ranks
Decode, batch 1	7.4–10.5 tok/s whenever tokens were flowing
First request after load	prefill tok/s is a cold-kernel lie (we saw ~1 tok/s on 56 tokens)

Four full loads and many generates on the same boot; no JACCL
tbt_post_recv SEGV, no LOC_PROT_ERR mid-run, no Metal 60 s watchdog, no
multi-chunk prefill deadlock. That last item is the contrast with Kimi-K3 on
the same fabric.

Prefill (unique prompts, cache-busting)

Prompt tokens	Prefill tok/s	Wall (incl. 8 decode tokens)
2,173	205	11.5 s
8,473	215	40.4 s
33,675	209	162 s
31,584–31,635 (needles / cache doc)	207–211	150–184 s

Linear through ~34k. If it holds: ~21 min to fill the native 262k window,
once. A unique ~63k-word prompt is in flight as this is written.

After the 34k prefill, the next eight greedy tokens were still coherent
English. Long decode is where quality died, not long prefill.

Prefix cache (measured)

Frozen ~31.6k-token document (planted site record + unique filler). First
user turn cached; each question is a second user turn so the prefix is a
token-level prefix. GDN cannot trim, so each hit clones the snapshot
(mx.array copy) and extends only the suffix.

Case	Prompt seen by generate	Wall	Result
miss (full doc + “what is the case ID?”)	31,635 tok @ 207 tok/s	161 s	closed </think>, answer Q38-7741
fill (doc only, no decode)	31,613 tok @ 211 tok/s	150 s	cache 894 MB, can_trim=False
hit, same ID question	22 suffix tok, clone 0.02 s	8.4 s	closed, Q38-7741
hit, date / amount / valve / site	22 suffix tok	31–35 s	thinks about the planted record, then junk to the 256 cap

The air-gap shape is real on the speed side: one 150 s fill, then ~8 s
for a short greedy question instead of another 160 s. Cache is small
(0.89 GB at 32k; linear that is a few GB at 128k). Quality still dies as
soon as the think is longer than a peaked lookup.

Greedy short tasks (not evals)

Probe	Closed </think>	What came out
81 × 47, 256 max	yes	3807, 70 tok @ 8.0 tok/s
384 + 769	no	think contains 1153, then a repeating junk token to length
remainder 1000 / 7	yes	6, 70 tok
extract planted case ID from a short paragraph	yes	Q38-7741, 57 tok
“is the valve open?” (it is closed)	yes	No then one junk token
four-field JSON from a one-line record	no	starts to parse, then junk to length

17 × 23 on an earlier load was the same pattern: greedy low-effort 391
and a close, 65 tok @ 8.5 tok/s.

Official sampler (same trivial math)

Sampler	Result
card temp=1.0, top_p=0.95, top_k=20	1024 tok of junk, never closed
temp=0.6 / 0.8	same collapse
temp=1.0, presence_penalty=0.5	multilingual junk, no close
temp=1.0, presence_penalty=1.0	7 tok, “answer” <tool_call>9

Needles in a ~32k haystack

Planted sentence The secret token is WATTLE-5519. at start, middle, or end
of unique filler. Greedy, 256 new tokens.

Prefill stayed 208–209 tok/s. The think named the right region (start /
between paragraphs / near the end) on all three. None closed, none emitted
the token: the same junk token took over. Long prefill attends. Decode
still dies.

The repeating junk token on this pack is 资源篮. It shows up in sampled
decode, long greedy, and any think that runs past a few dozen tokens.

We have no perplexity, no held-out loss, no coding/agent bench. We know
this 4-bit pack can emit a correct short greedy answer when the mode is
peaked, and that every sampled setting we tried, plus every longer think,
destroyed the tail. That is not a capability profile.

---

Reading the quality hole

The MoE body at 4-bit is why the model fits and why TP4 prefills at ~210 tok/s.
The lm_head is a 248k-way classifier quantized the same way. Argmax only
needs the mode; short arithmetic is peaked, so greedy can still pick 391 or
3807. Sampling (top_k=20, top_p=0.95, temp≥0.6) needs a calibrated
neighbourhood. We believe 4-bit group-64 noise is enough to mix junk tokens
into that neighbourhood; one bad sample then locks the decode. Long greedy is
the same story with more near-ties.

mlx-lm mixed recipes already special-case lm_head to higher bits. A cheap
follow-up is in progress: 8-bit lm_head and embed_tokens, body still
4-bit. Official FP8 stores those two tensors as BF16 already
(248320 × 8192). Staging them to affine 8-bit group-64 took 2.0 s and
4.3 GB. They have not been swapped into the live replica yet (the current
load still holds the 4-bit shard). Until that run exists, “4-bit Qwen3.8 on
Apple” should not be described as the model.

---

What this can achieve, honestly

Stable TP4 hosting of a 2.4T 4-bit replica on four M3 Ultras, ~8–10 tok/s
decode, ~210 tok/s chunked prefill through 34k, linear so far, and a working
prefix-cache extend (894 MB at 32k, miss 161 s → hit 8.4 s). That is a
real MLX/JACCL result. Kimi-K3 on the same mesh was not a usable runtime.

It is not a chat model, not Max, and not a capability claim. 8 tok/s plus
forced think is overnight / queue latency. Independent jobs each pay full
prefill unless the prefix is frozen. Concurrent multi-stream is untested.

The remaining use case we take seriously is air-gapped text that must not
leave the building, with a large frozen prefix (corpus, case file, tree)
and short greedy questions, decode capped. Cost-per-token versus an API
does not apply if there is no API you are allowed to use. The speed half of
that story is now measured. The quality half is only peaked lookups
(Q38-7741, 3807, 391). Harder questions against the same prefix start a
real think and then loop.

If the data can go to Qwen Cloud or a GPU cluster, it should. FP8/BF16, a
full-precision head, MTP, and Max’s product features live there. Four Mac
Studios are a serious amount of money to generate eight tokens a second of
unevaluated 4-bit output.

We would not retire 512 GB Studios. They are excellent at 30B–300B, one node,
current mlx-lm / omlx. We would treat trillion-class serving on four
consumer machines as a closed experiment unless the 8-bit head run changes
the quality picture.

That is the experiment. A year ago a 1T model was frontier. Capability has
moved: for most work a much smaller model in a competent harness is as good or
better, and it is the smart default. The largest weight is no longer necessary
for the majority of tasks. We already measured a version of this on the same
four machines: one Studio on a ~300B-class model with a fast local runtime
beat four Studios on a 1.6T checkpoint in a reference graph, on both speed and
the jobs we cared about. Topology helps a dispatch-bound runtime; kernels and
the right-sized model help more.

The question that remains is when a multi-trillion model is necessary
enough that waiting minutes per answer is warranted. That buyer is not
shopping for Thunderbolt cables. They are paying for data centre equipment
worth millions (or renting it) because they need FP8/BF16, a full-precision
head, MTP, and a harness that matches the card. The fact that four second-hand
M3 Ultras can host 2.4T at reading speed is remarkable. It is also the
limit of what is viable on consumer metal. Past that limit, the work and the
capital belong in a hall with real interconnect, not a desk with TB5.

---

Still running, and what we will post next

On this boot, next:

• unique ~64k then ~128k prefill (linearity / JACCL / peak RAM)
• swap the staged 8-bit head + embeddings into the live replica
• re-run: greedy canary, official sampler on 17 × 23, one long greedy, one
  prefix-cache hit

We are not claiming those results in this post. Happy to take questions in
the meantime: convert layout, GDN cache, prefill_step_size, or the quality
hole.

---

Reproduction sketch

# once per boot — do not run distributed_config twice
mlx.distributed_config --over thunderbolt --backend jaccl --auto-setup \
  --hosts n1,n2,n3,n4 --output-hostfile hostfile.json
# hostfile envs: MLX_METAL_FAST_SYNCH=0

# ranks: same replica path on every node (mlx_lm.share, then reboot if PDs are spent)
mlx.launch --hostfile hostfile.json ./bench.py

generate / stream_generate(..., prefill_step_size=512). Chat template:
reasoning_effort in {xhigh, medium, low} (default xhigh); do not pass
enable_thinking=false. Wired limit = max_recommended_working_set_size.
Prefix reuse: prefill the common turn into make_prompt_cache, clone, pass
only the suffix as prompt with prompt_cache=clone. Do not replay the
full prompt into the same cache (generate_step does not skip a cached prefix).

Happy to share the streaming converter and the one-load bench harness if useful.

---

What we are not claiming

• That this matches Qwen3.8-Max, or any published Qwen3.8 number.
• Perplexity, task scores, or “it reasons well.”
• 64k, 128k, 262k, or 1M context as executed (34k is the longest prompt we
  have finished; 63k is in flight).
• That 8-bit heads fix decode. They are staged, not loaded.
• MTP, speculative decode, or multi-user serving.
• That 4-bit is a good quant for this model: only that it is the quant that
  fits.

Possible, on this metal, is not the same as worth it.

[guruswami-ai]

Update (same boot): 126k prefill, then 8-bit heads

Prefill ladder finished after the first post. Unique prompts, 8 greedy tokens, prefill_step_size=512:

Prompt tokens	Prefill tok/s	Wall
63,075	201	315 s
126,077	188	672 s

Gentle roll-off from ~209 tok/s at 34k. Fabric stayed quiet. 262k still unrun.

Then we swapped 8-bit lm_head + embed_tokens (official BF16 → affine 8-bit group-64, 4.3 GB, 2 s to stage) and left the MoE body at 4-bit. New load: 192 s, 344 GB active / 367 GB peak.

Probe	4-bit head	8-bit head
greedy 17 × 23	closed, 391	closed, 391
card sampler 17 × 23	1024 tok junk	1024 tok junk
greedy 384 + 769	1153 in think, then junk	same
greedy JSON extract	starts, then junk	same
prefix-cache hit, planted case ID	8.4 s, Q38-7741	8.6 s, Q38-7741

The cheap head fix is not enough. Short greedy still works; sampled and long greedy still collapse to the same junk token (资源篮). Either the 4-bit body already poisons the neighbourhood, or something else in this pack does. An 8-bit body does not fit on four × 512 GB, so we cannot separate those causes here.

[guruswami-ai]

Update: batch

BatchGenerator on this hybrid (69× GDN ArraysCache + 23× FA) survives TP4. Greedy only, one inserted batch, not continuous serving.

First B=1 call paid a 330 s cold kernel. After that, aggregate decode scaled 7.9 → 14.0 → 24.3 tok/s at B=1/2/4 (per-stream ~7.9 / 7.0 / 6.1). Unique ~2k prefills at B=4 were 322 tok/s aggregate vs ~210 single-stream. Peak stayed 367 GB.

Quality unchanged. The useful “batch work” shape is still a frozen prefix plus short greedy questions, not a bigger microbatch.

[guruswami-ai]

Theory: why decode falls into 资源篮

资源篮 is not EOS and not in the chat template. It is ordinary BPE id 107300, a single merge for 资+源+篮 (“resource basket”, the cart on Chinese edu / download sites). The other junk is the same class of one-id web-chrome phrases (本文为博主, 转载请附上, 点击复制链接, 趣阁, $fdata). So the loop is a crawler-boilerplate attractor, not a reserved stop token we mis-decoded.

What we know:

• Prefill attends (32k needles name the right region). Fabric is stable through 126k.
• Short peaked greedy survives (17×23 → 391, closed </think>).
• Official sampler (temp=1.0, top_p=0.95, top_k=20) and any long think die.
• 8-bit lm_head + embed_tokens from official BF16 did not change that. Router gates were already 8-bit.

So the 248k classifier is probably not the hole. The residual arriving at the head still has the right mode on peaked tasks and a destroyed neighbourhood otherwise. One near-tie picks a footer token; after that the prior in that mode is almost only more footer tokens.

Leading guess: 4-bit MoE experts. Most of the pack is SwitchGLU stacks at affine 4-bit group-64. Routing can still be sharp (8-bit gates) while the chosen expert is noisy. Argmax of a spike works; top_k=20 and long greedy do not.

Runner-up: a systematic convert error in the same place (FP8 block-128 dequant, expert index when stacking into SwitchGLU, conv1d axis). That would look identical and we have no teacher layer to rule it out.

Third: GDN decode drift. 69/92 layers are Gated DeltaNet. Prefill of 126k still yields eight coherent tokens, so chunked prefill is not dead. Failure is sequential decode after a good start. If ArraysCache is slightly wrong on N=1, error would accumulate exactly on long think and spare the 65-token canary.

We cannot split these. Official FP8/BF16 does not fit on 4×512 GB, so we have no PPL or same-prompt teacher. If anyone has Qwen3.8-2.4T on GPU at FP8/BF16 (or a 4-bit pack from a different convert), a useful check is: does greedy 384+769 stay on the rails past the first 80 tokens, or does it also fall into 资源篮 / 本文为博主? If the full model never does this, it is our pack (quant or convert). If it does, 4-bit only made a real mode sticky.

Questions for mlx-lm: is qwen3_5_moe + GDN ArraysCache believed numerically safe for long N=1 decode after a multi-chunk prefill? And is remapping qwen3_5_moe_text → qwen3_5_moe the intended load path?

[guruswami-ai]

Update: hosted Qwen3.8-Max does not emit the junk string

Same three prompts that die on the MLX 4-bit pack, run against hosted qwen3.8-max (thinking on, temp=1.0, top_p=0.95; API has no top_k):

Prompt	MLX 4-bit TP4	Hosted Max
384 + 769	think has 1153, then 资源篮 loop	think + 1153, finish=stop
17 × 23 (card sampler)	1024 tok junk	think + 391, stop
JSON extract (Q38-7741…)	starts to parse, then junk	valid JSON, stop

Zero occurrences of 资源篮, 本文为博主, 转载请附上.

Max is not the open Qwen3.8-2.4T-A95B checkpoint, so this is not a bit-exact teacher. It does show that the hosted 2.4T-class product does not live in that crawler-footer mode. Combined with 8-bit lm_head changing nothing, we now think the hole is on the MLX side of our pack: 4-bit SwitchGLU experts, the streaming FP8 convert (mx.from_fp8 / expert stack / conv1d axis), or GDN ArraysCache on long N=1 decode after chunked prefill. “Qwen3.8 always dumps blog chrome” looks unlikely.

A GPU run of open FP8/BF16 on those same prompts would still split convert-vs-runtime. If anyone has that, we would like to know whether greedy 384+769 stays clean past ~80 tokens.