V0.3.0/phase m—apple m series complete maximally fast qwen backend - #100
Open
thanos wants to merge 13 commits into
Hidden character warning
The head ref may contain hidden characters: "v0.3.0/Phase_M\u2014Apple_M-series-Complete-Maximally_Fast_Qwen_Backend"
Open
V0.3.0/phase m—apple m series complete maximally fast qwen backend#100thanos wants to merge 13 commits into
thanos wants to merge 13 commits into
Conversation
Why Prefill (large GEMMs) and decode (matvecs + KV + launches) need separate metrics before M2/M3 optimization; a single tok/s number hides the regimes. What - CLI: `qwen-bench` [--mini] with human table + json; `stageM1` ledger - build.zig: stageM1 + qwen-bench steps (and integration hooks) - Gpu: total_encodes / total_waits + resetLaunchCounters - Session.generate: snapshot Metal launches for prefill and each decodeToken - Arch helpers: GEMM shape strings, decode B/tok estimate - run/chat footer: prefill_tok_s, decode_ms_per_tok - Docs: M1 stage + tutorial 15; stageM1-dev-laptop mini + full-model numbers - M0 docs/ledger: baseline landed, gate checklist still open Measured (dev laptop, full qwen3-0.6b, 17 prompt tok, max-tokens 2) apple prefill ~14 tok/s; decode ~2149 ms/tok; enc/tok = wait/tok = 476 closed #92
…th, roofline, JSON - `stageM2` ledger + `zig build qwen-profile` - Family timing + `qwen.*` signposts (`ZYNFER_SIGNPOSTS=1`) on the Qwen path - Docs: stage M2, tutorial 16, `bench/results/stageM2-dev-laptop.md` 1. **LM head + final norm (~53%)** — still CPU tied matvec 2. **MLP (~22%)** — Metal, M0 per-op encode+wait 3. **QKV (~12%)** — same launch tax Also: **476** encodes=waits/tok; STREAM ≈ **228 GB/s**; roofline ≈ **96 tok/s** ideal → measured ≈ **1%** of that. Est. launch overhead ≈ **122 ms** (inside Metal rows). ```bash ./zig-out/bin/zynfer qwen-profile --mini ./zig-out/bin/zynfer qwen-profile models/qwen3-0.6b.zynfer --prompt "Explain gravity simply." ``` closed #93
closed #94 ** Results (full Qwen3-0.6B) ** | Metric | M0 baseline | M3 batched | Speedup | |--------|-------------|------------|---------| | Prefill | ~1360 ms | **177 ms** | ~7.7× | | Decode | ~2640 ms/tok | **231 ms/tok** | ~11.4× | | Waits/tok | 476 | **2** | 238× fewer | ** What landed ** - `src/backends/apple/qwen_schedule.zig` — resident weights/KV, one CB for all layers + one for final norm/LM head - Retained: `silu_mul`, `add_rmsnorm_f32`, Metal LM-head matvec - Rejected/deferred: Q/K+RoPE fuse, attention tiling, dequant→M5, ICB - A/B: `ZYNFER_QWEN_METAL=baseline` - Docs: stage M3, tutorial 17, `bench/results/stageM3-dev-laptop.md` ```bash ./zig-out/bin/zynfer qwen-bench models/qwen3-0.6b.zynfer --prompt "Explain gravity simply." --max-tokens 2 ZYNFER_QWEN_METAL=baseline ./zig-out/bin/zynfer qwen-bench --mini --max-tokens 4 ```
…nts the attention `kv_len` cap as **2048** (via `max_attention_kv`), but the test still looked for `"256"`. Updated `tests/integration/cli.zig` to assert on `"attention kv_len cap"` and `"2048"` instead. `zig build integration -Dhip=off` passes now.
**Half-precision Metal path** for Qwen (`ZYNFER_QWEN_METAL=bf16|half|fp16`): - **bf16 resident weights** on GPU (all projections, norms, embed, LM head) - **bf16 KV cache** (append from f32 Q/K/V; attention widens to f32 for dot/softmax) - **f32 activations** and f32 accumulators in GEMM, RMSNorm, and attention - M3 schedule preserved (~2 waits/forward) Default remains M3 f32 (`batched_resident_kv_fused`). | Area | Files | |------|--------| | Kernels | `src/backends/apple/kernels.metal` — `matmul_bf16_f32`, `matvec_bf16_f32`, `rmsnorm_bf16_f32`, `add_rmsnorm_bf16_f32`, `kv_append_bf16`, `attention_f32_bf16_kv` | | Ops | `src/backends/apple/ops.zig` — encode*Bf16, `matmulF16`/`matvecF16` | | Schedule | `src/backends/apple/qwen_schedule.zig` — `MetalStack.half_mode`, path `batched_resident_kv_bf16` | | Upload | `src/runtime/bf16.zig` — `encodeFromF32` | | Estimates | `qwen3.estimateDecodeBytesPerTokenHalf` (≈½ f32 bytes/tokenge ```bash zig build test -Dhip=off zig build integration -Dhip=off zig build stageM4 -Dhip=off ZYNFER_QWEN_METAL=bf16 ./zig-out/bin/zynfer qwen-bench --mini --max-tokens 4 ``` - Mini bf16 logits vs CPU: **5e-3 atol** (dtype-justified) - f32 batched path: still **3e-3** - Caps: `fp16`/`bf16` enabled on Apple - Docs: `docs/stages/M4-half-precision-metal.md`, `docs/tutorials/18-half-precision-inference.md`, `bench/results/stageM4-dev-laptop.md` Fill in measured prefill/decode numbers in the bench ledger on your machine with the full model. **M5** (int8 session weights) is next. closed #95
Close M4 gate: greedy parity, native bf16 GPU upload, full-model A/B. closed #95 Why M4 kernels/schedule landed earlier, but the gate still needed (1) greedy token parity vs CPU, (2) artifact→GPU half upload without f32 promote, and (3) measured full-model A/B + roofline characterization. What - Native half upload: `qwen_weights.copyArtifactToBf16` + schedule `copyWeightHalf` (bf16 memcpy / f16→bf16; transpose in bytes; f32 fixtures still narrow from host) - Runtime: `float16.zig`; bf16 `fromF32`/`encodeFromF32` - Converter: dtype summary on stderr (`dtypes: bf16=N`) - Tests: mini greedy CPU vs bf16 Metal; full-model greedy behind `ZYNFER_FULL_MODEL_TESTS=1` (slow CPU oracle) - Docs/ledger: numerics, apple-backend, artifact-format, tutorial 14/18, stage M4, checkpoint README; filled `stageM4-dev-laptop.md` Measured (qwen3-0.6b, 17 prompt tok, max-tokens 2) | Path | prefill_ms | decode_ms/tok | B/tok_est | | f32 | ~214–261 | ~231–234 | | | bf16 | ~165–170 | ~252–254 | 1.19e9 | Prefill ~1.5×; decode flat (still encode-bound ~590 enc/tok). Roofline ideal tok/s doubles; measured fraction drops (~0.105 → ~0.045). Retain bf16 for footprint + prefill; decode win deferred to M5 kernels. ```bash ZYNFER_QWEN_METAL=bf16 ./zig-out/bin/zynfer qwen-bench \ models/qwen3-0.6b.zynfer --prompt "Explain gravity simply." --max-tokens 2 ZYNFER_FULL_MODEL_TESTS=1 zig build test -Dhip=off ``` ```
Why M4 kernels/schedule landed earlier, but the gate still needed (1) greedy token parity vs CPU, (2) artifact→GPU half upload without f32 promote, and (3) measured full-model A/B + roofline characterization. What - Native half upload: `qwen_weights.copyArtifactToBf16` + schedule `copyWeightHalf` (bf16 memcpy / f16→bf16; transpose in bytes; f32 fixtures still narrow from host) - Runtime: `float16.zig`; bf16 `fromF32`/`encodeFromF32` - Converter: dtype summary on stderr (`dtypes: bf16=N`) - Tests: mini greedy CPU vs bf16 Metal; full-model greedy behind `ZYNFER_FULL_MODEL_TESTS=1` (slow CPU oracle) - Docs/ledger: numerics, apple-backend, artifact-format, tutorial 14/18, stage M4, checkpoint README; filled `stageM4-dev-laptop.md` Measured (qwen3-0.6b, 17 prompt tok, max-tokens 2) ``` | Path | prefill_ms | decode_ms/tok | B/tok_est | | f32 | ~214–261 | ~231–234 | 2.39e9 | | bf16 | ~165–170 | ~252–254 | 1.19e9 | ``` Prefill ~1.5×; decode flat (still encode-bound ~590 enc/tok). Roofline ideal tok/s doubles; measured fraction drops (~0.105 → ~0.045). Retain bf16 for footprint + prefill; decode win deferred to M5 kernels. ```bash ZYNFER_QWEN_METAL=bf16 ./zig-out/bin/zynfer qwen-bench \ models/qwen3-0.6b.zynfer --prompt "Explain gravity simply." --max-tokens 2 ZYNFER_FULL_MODEL_TESTS=1 zig build test -Dhip=off ``` **Stage M5 is done.** Opt-in `ZYNFER_QWEN_METAL=int8|q8` runs per-row symmetric int8 projections with fused Metal dequant; 4-bit stays deferred. | Path | prefill_ms | decode_ms/tok | B/tok_est | | --- | ---: | ---: | ---: | | bf16 | 170 | 253 | 1.19e9 | | **int8** | **126** (~1.3×) | **119** (~2.1×) | **0.60e9** | Greedy quality proxy (`"Explain gravity simply."`, 2 tokens): CPU / bf16 / int8 all produce `Gravity is`. Mini logits PASS @ 5e-2; `wait/tok` = 2. ```bash ZYNFER_QWEN_METAL=int8 ./zig-out/bin/zynfer qwen-bench models/qwen3-0.6b.zynfer \ --prompt "Explain gravity simply." --max-tokens 2 zig build stageM5 -Dhip=off ``` Ledger: `bench/results/stageM5-dev-laptop.md`. Tutorial: `docs/tutorials/19-quantization-on-apple-silicon.md`. Both polish items are done. Fixed `quantize_zynfer_int8.py` (was missing TensorEntry’s 2-byte pad; now NumPy-backed). Wrote `models/qwen3-0.6b-int8.zynfer` (867MB, gitignored): - **197** projections quantized - **worst dequant abs err = 4.86e-3** (budget 0.05) Added `Stage M5: greedy tokens match CPU (full model when artifact present)` — same opt-in as M4. `ZYNFER_FULL_MODEL_TESTS=1 zig build test -Dhip=off` → **90 pass / 3 skip** (M4 + M5 full greedy both PASS). Ledger/tutorial/README updated. Still deferred: 4-bit, runtime load of on-disk i8. closed #96
…red by effort vs payoff: 1. **Longer mini alloc test (~30 min)** Extend the existing M6 test to run `generateCached` at `max_seq=32` with a 2-token prompt → ~28 decode steps. Matches fable-5 “long generation” without full-model CI time. 2. **Opt-in full-model long generate test (~1 hr + slow run)** Mirror M5: `ZYNFER_FULL_MODEL_TESTS=1` test, greedy, e.g. 32 tokens, `FailingAllocator` flat after warm-up. Good for local/CI opt-in, not default `zig build test`. 3. **Streaming buffer pre-capacity (~1 hr)** In `run`/`chat`, `pending.ensureTotalCapacity(max_new_tokens * 4)` (or similar) before `generate`. Cuts per-token realloc in the CLI; engine path already static. 4. **Ledger refresh (~15 min)** Re-run ITL with `--max-tokens 32` or `64` and add a row to `bench/results/stageM6-dev-laptop.md`. Documentation only. 5. **`mem-report` for bf1** One command each, note metal weight bytes vs f32 in the ledger. Validates M4/M5 + M6 together; no code unless you want path labels in JSON. 6. **Stronger ITL in `qwen-bench`** Optional `--max-tokens` default bump for a dedicated M6 bench snippet. Small CLI/doc change. | Item | Why wait | | --- | --- | | **Skip host f32 weight twin** | Needs Apple-only load → GPU-only resident weights; breaks CPU oracle unless you split sessions. Large RSS win, large refactor. | | **Baseline path host twin skip** | Baseline still runs `qwen_block.forward` on host tensors per op — needs scratch/KV. Can’t mirror batched without a new baseline schedule. | | **GPU heap assertion** | Needs Metal `allocShared` instrumentation or bridge hooks; host assert is the right gate for M6. | | **On-disk i8 without host pack** | M5 polish, not M6. | **#1 + #3 + commit** — stronger gate proof + cleaner streaming, ~2 hours, no architecture risk. **# if you want parity with M5’s full-model opt-in story. Want me to implement 1+3 (or the full set 1–4)? **M6 polish items 1–6 are done.** `generateCached` at `max_seq=32` → **30 decode tokens**, `FailingAllocator` flat after warm-up + reset. `ZYNFER_FULL_MODEL_TESTS=1`: **32 greedy tokens** on qwen3-0.6b, flat after warm-up. `run`/`chat`: `out_ids.ensureTotalCapacity(max_new_tokens)` and `pending.ensureTotalCapacity(max_new_tokens × 8)` when streaming. `bench/results/stageM6-dev-laptop.md` updated with ITL at **32** and **64** tokens (bf16). @ `max_seq=128`: | Path | metal_weights | peak_rss | | --- | ---: | ---: | | f32 | 2.38e9 | 6.29e9 | | bf16 | 1.19e9 | 5.10e9 | | int8 | 1.22e9 | 5.13e9 | - Default full-model `max_new`: **8 → 16** - Per-backend **ITL p50/p95/p99** + JSON `itl_*` fields - Apple example: p50≈131 ms, p95≈138 ms @ 16 tokens **Tests:** 97 pass / 6 skip · integration green. Optional local check: `ZYNFER_FULL_MODEL_TESTS=1 zig build test -Dhip=off` (~6 min). closed #97
Tutorial 21 — plain-English verdict; why (a)/(b) were skipped (not “raced and lost”); unified memory ≠ free Metal↔Core ML handoff; “NInfer for Core ML” vs zynfer; what M8 delivers next. Stage docs — M7 rationale aligned with that; new docs/stages/M8-apple-capstone.md (Apple-complete gate, what M8 is / isn’t). Ledger + roadmap + apple-backend — skip wording, handoff note, M7/M8 gates, milestone J clarified as larger Metal checkpoint (not ANE). zynfer stageM7 CLI text — same story so the live ledger matches the docs. Closed #98 - Stage M7 with a final Core ML/ANE reject at Qwen scale. Keep Metal as the Apple path, document why prefill/MLState were skipped, and add toy load smoke plus an xctrace recipe without clearing retain.
Declare Backend 1 (Apple) complete: prove the Metal engine on a model people actually run, with reproducible numbers and no silent 0.6B assumptions. Registry and architecture - Add src/model/registry.zig (HF repo, paths, dims, artifact version, int8 SHA). - Register Qwen3-4B (h=2560, 36 layers, GQA 32/8); validate full-vocab loads. - Raise .zynfer mmap size limit to 32 GiB for bf16 4B artifacts. Runtime and tooling - Dequant on-disk i8 + .qscale to host f32 in qwen_weights (Metal then re-packs). - Converter --model-id / dim inference; setup_qwen.py --model 4b --quantize. - CLI: stageM8, setup --model/--quantize, tokenizer via registry HF dirs, chat default prefers models/qwen3-4b-int8.zynfer when present. Docs and verification - Ledger apple-capstone-dev-laptop.md (0.6B + 4B matrix, KV budget, haiku example, why slower than Ollama). - Tutorial 22; M8 stage/roadmap/README; zig build stageM8 + integration test. Not in this commit: i8→Mel without the host f32 twin, matched llama.cpp/MLX tok/s numbers, or Phase R/S. - closed #99 - closed #101
…l — no host f32 proj twin. **What changed** - `Weights.loadForAppleQ8` — embed + norms only when artifact is i8 - `LayerQ8Weights` — prefer artifact i8+`.qscale` upload (HF `[out,in]`); fall back to pack-from-host - `Session.initWithBackend` — slim path when `apple` + `ZYNFER_QWEN_METAL=int8` + i8 artifact **Measured (4B int8, `mem-report` max_seq=512)** | | Before | After | |---|---|---| | host_weights | ~16.1 GiB | ~1.45 GiB | | peak_rss | ~26 GiB | ~10.8 GiB | | metal_weights | ~5.6 GiB | ~5.6 GiB | Chat still works (~3.8 tok/s). Host leftover is mainly the embed f32 used to pack tied `lm_head` Q8. Ledger + tutorial 22 updated. closed #103
- closed #106 ``` Drop Apple int8 host embed twin and switch Q8 KV to bf16. Finish the M8 residency debt after i8→Metal projection upload: skip host f32 embed/lm_head on loadForAppleQ8, gather from Metal-resident bf16 embed, use bf16 matvec for tied lm_head, and run bf16 K/V on the int8 path. Cuts 4B int8 host_weights to ~0.75 MiB and peak RSS to ~8.3 GiB; refresh the capstone ledger with long-prompt/cold-warm numbers. ``` Slightly longer body if you want more detail in the commit: ``` Drop Apple int8 host embed twin and switch Q8 KV to bf16. After projection i8→Metal upload, the remaining twin was the ~1.45 GiB host f32 embed used for gather and tied lm_head packing. This change: - loadForAppleQ8: norms only (embed_resident=false) - MetalStack: bf16 embed table + forwardLastLogitsFromTokens - tied lm_head: bf16 matvec on Scratch.embed (no vocab-wide i8 pack) - Q8 path: bf16 KV via encodeKvAppendBf16 / encodeAttentionBf16Kv - update mem-report accounting, stagKV budget text, capstone ledger (long prompt 0.6B/4B, cold/warm, RSS), and tutorial 22 Measured on qwen3-4b-int8 (max_seq=512): host_weights ~0.75 MiB, peak_rss ~8.3 GiB, chat still ~3.9 tok/s short prompt. ``` Implemented the optional polish items that were feasible in-engine; matrix cells are filled in the ledger. **Engine changes** 1. **No host embed twin** — `loadForAppleQ8` skips embed/lm_head; gather from Metal-resident bf16 table via `forwardLastLogitsFromTokens`. 2. **bf16 KV on int8 path** — Q8 uses `encodeKvAppendBf16` / `encodeAttentionBf16Kv` (half KV traffic). 3. **Tied lm_head** — bf16 matvec on embed table at decode (no vocab-wide i8 pack at init — that was ~2 min load). **Measured (4B int8, max_seq=512)** | | Before | After | |---|---|---| | host_weights | ~1.45 GiB | **~0.75 MiB** (norms only) | | peak_rss | ~10.8 GiB | **~8.3 GiB** | | load wall | ~minute-scale | **~20 s** | Chat still works (~3.9 tok/s short prompt). ** Ledger filled (`bench/results/apple-capstone-dev-laptop.md`) ** - Long prompt 0.6B/4B (213 tok) - Cold vs warm 4B - Updated memory + KV budget (bf16) - 4B bf16/f32 rows: **skipped** (7.5 GiB artifact too heavy for laptop) - llama.cpp/MLX: **still no Qwen3-4B GGUF; `mlx_lm` absent) **Deferred (documented, not implemented)** - **4-bit / denser quants** — needs new scheme + kernels - **Kernel polish** — attention/GEMV fusion beyond M6 - **Matched llama.cpp row** — needs GGUF conversion + fair A/B script Verify: ```bash zig build ZYNFER_QWEN_METAL=int8 ./zig-out/bin/zynfer mem-report models/qwen3-4b-int8.zynfer --max-tokens 512 ZYNFER_QWEN_METAL=int8 ./zig-out/bin/zynfer chat models/qwen3-4b-int8.zynfer "give me a haiku on snow" ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.