perf(qsa): multi-threadgroup scoring, quantized pooled keys, and deep MTP throttling for 546k+ decode speed - #413
Open
maceip wants to merge 718 commits into
Open
Conversation
The default-on ~/.mtplx/logs/request-log-<port>.jsonl claimed 'numeric/hash telemetry only — no prompt or completion content' but every record carried request_last_user_preview with literal user text, rotating across 4x64MB generations of durable history. Redact at the durable sink only: the JSONL line now carries a stable sha256:<16-hex> digest under the same key (turn correlation survives), while the in-RAM dashboard ring and flight-recorder trace labels keep the literal preview for live diagnosis. The flight recorder's terminal event already copies a curated numeric set, so nothing else content-bearing reaches disk. MTPLX_REQUEST_LOG_CONTENT=1 opts back into literal previews for local debugging. Help text and default-path comment updated to match reality.
…vention per sidecar (youssofal#301) Forge's embedded-head extraction ran every tensor through sanitize_plain_weight, whose MTP_RMSNORM_ALWAYS_SHIFT_SUFFIXES branch added +1.0 unconditionally to q_norm, k_norm, and mtp.norm.weight. Correct for HF-native Qwen3.5/3.8 exports (zero-centered norms), it corrupted exactly those three tensors when the source already stored absolute gains (MLX- converted checkpoints like youssofal#301's) — the drafter then mismatches its trunk and acceptance collapses to 0-2%, making every depth slower than AR. Measured on the real fleet (shipped 3.8/4B/9B sidecars vs raw HF Qwen3.5-4B embedded head): the low set separates delta vs absolute at 0.30-0.39 vs 0.87+, q/k at 0.73-0.75 vs 1.73+, but the FINAL norm overlaps across conventions (raw-delta 4B mean 2.58 vs absolute 3.8 mean 2.25) — it can never be judged per tensor. So the convention is now decided once per sidecar (both separable families must agree: max qk < 1.25 AND min low < 0.5, the exact two-signal gate the runtime heal shipped for youssofal#176) and applied to all seven norms, in one shared shift_delta_mtp_norms(): - forge embedded extraction: sanitize per tensor, then one ensemble shift - AWQ/compressed-tensors convert: same shift before mtp.safetensors write - runtime heal (_heal_raw_delta_mtp_norms): collapses to the shared call Absolute-convention sources now pass through byte-identical. Tests pin both conventions, both key namespaces, the overlap-final-norm case, and the missing-family conservative default.
…ng image parts in user canonicalization (youssofal#327) The agent-transcript canonicalizer (active whenever tools are present) pooled consecutive user messages by stringifying both sides through _content_to_text, so an OpenAI image_url part in either message was silently discarded before the vision extractor ran — DSH's standard preset (user(image+text) -> user(runtime snapshot)) always answered blind, while its minimal preset worked. Likely the same family as youssofal#328 (Pi + vision: Pi sends tools, MTPLX chat does not). Guard: a message whose structured content carries any non-text part (image_url, audio, unknown parts) is exempt from tandem-repeat collapse, duplicate dropping, and consecutive-user merging — it is appended as-is. Consecutive text-only users keep merging (covered by regression test), and downstream already renders consecutive user turns (the skipped_repeated_assistant branch has always emitted them).
…ode and refuses --stock-ar loudly (youssofal#285) The bench --harness depth-sweep branch accepted --stock-ar, --generation-mode, --depths, and --seed, then discarded all four (depths='3', seed=0, compare_ar=False hardcoded) — argparse validated, exit code was 0, and a result file was written, so four 'different' A/B configs produced byte-identical MTP-D3 runs and an 'AR baseline' that drafted 129 tokens. Now: --depths and --seed thread straight through (defaults unchanged: '3'/0), --temperature/--top-p/--top-k and explicit --draft-* overrides are honored over the runtime-contract values, and --generation-mode ar maps to compare_ar+ar_only (a real target-only AR baseline; depth<1 already fails loudly in the runner). --stock-ar cannot be delivered by this harness (it always loads the MTP runtime), so it exits with a pointer to --harness direct-http instead of pretending. The result profile records the requested depths and ar_baseline instead of a hardcoded depth: 3.
…identifier fallback ladder (youssofal#329) The Hardware card guessed the chip from the Mac model identifier with a table that was off by a generation across the board (mac14->'M3', mac16->'M3 Ultra', mac13->'M2'), so a Mac Studio M2 Max (Mac14,13) read 'M3'. The server has always sent the true sysctl brand string in the same payload (machine.chip: 'Apple M2 Max') — the dashboard just never used it. The badge now prefers the reported chip (Apple prefix stripped); the identifier ladder remains only as a fallback for a missing brand string and now maps chip families correctly: Mac13=M1-era Studio, Mac14=M2 (incl. Studio M2 Max/Ultra), Mac15=M3, Mac16=M4, Mac17=M5. Static bundle rebuilt (tsc -b clean; new logic verified present in the emitted asset).
… replacing the link (youssofal#292) The terminal-command installer appended its PATH block by rewriting ~/.zshrc with an atomic write — a temp-file rename that replaces a symlink with a plain regular file. On the very common dotfiles setup (stow, chezmoi, yadm, hand-rolled repos symlink ~/.zshrc into a git repo) this silently detached the live file from version control: the repo copy and the real file diverge with no signal, and the next dotfiles sync clobbers the divergence. Resolve the symlink and write to the target, so the PATH line lands inside the dotfiles repo and the link survives. Regression test pins the link destination, the appended line in the target, and the preserved content.
…ch; bf16 keeps tuned 512 (youssofal#319-derived) grzracz reported (PR youssofal#319) that the gdn_capture fused_add_rmsnorm call at threadgroup_size=512 perturbs prefill above 64 rows. Probed on M5 Max / MLX 0.32: CONFIRMED for fp16 — max|diff| up to 3.9e-3 vs the unfused x+r / mx.fast.rms_norm reference once the grid crosses 2^15 threads (rows > 64 at 512 lanes), data-dependent by seed. bf16 is bit-exact at 512 in every probe, and the default 1024-lane loop is bit-exact for both dtypes at every probed shape. fp16 is exactly the M1/M2 model lane (-FP16 siblings), so this was a silent numerics leak on the hardware least able to absorb it. Fix keeps the tuned 512 width where it is provably exact (bf16 — mainline path byte-identical in output and kernel config) and routes fp16 to the exact default dispatch. No env knob (PR youssofal#319's dial declined in favor of the dtype guard); exactness contract pinned by tests across seeds. Follow-up for the gated bench pass: confirm fp16 prefill TPS is flat under the 1024-lane loop.
…ussofal#330 by @CharliePetch) Adopted as-is: completes the bit-exact replay envelope (prompt_token_ids at dispatch + raw sampled output ids at finalize) for the opt-in request capture. Zero hot-path cost (inside the capture_dir() gate), never-raises coercion verified on ints/None/garbage; serial, ar_batch, and mtp_batch finalizers all carry the field. Local pytest: test_request_capture green.
…iftedx) Adopted as-is: a module_overrides entry with quantize:false now returns False from the quantize_model predicate so sensitive modules keep their source precision, instead of being silently 8-bit-quantized by the fallback params. Typed override tuples, PR's own regression test included; test_forge_mixed_convert green locally.
…ussofal#273 by @El-Patronum) Adopted with a comment cleanup: cmark returns raw HTML as literal text, so <br> in a table cell rendered verbatim and kept the cell one unbreakable token that overflowed its fixed column into the neighbour. Single choke-point regex converts <br>/<br/> (case-insensitive) to real newlines before AttributedString parsing; comparison operators are untouched and the parse-failure fallback uses the same converted text. swift build + render tests green.
…active-read compactor (youssofal#282)
The May-era agent-rewrite machinery silently mutated client transcripts:
four content compactors truncated tool results (Kilo's active-read path
had no opt-out at all), heuristic text sniffs dropped client assistant
messages and hid declared tools (the Hermes 'direct reply turn: tools
are unavailable' report), the Pi convergence contract told the model to
stop reading files after 14 tool results, and any client hinting 'pi'
had its stream cut after the first complete tool call even though Pi
executes every tool call in a turn (pi-agent-core ships both
executeToolCallsSequential and executeToolCallsParallel).
Community receipts drove the default: tcpdump proof that Pi sends the
full file and the model receives a digest, a 10-run task measurement at
8/10 success with compaction on vs 10/10 off with no latency benefit,
and Hermes skill files silently reduced to fragments.
The new contract:
- Unset (default): passthrough. No content compaction, no steering
contracts, no heuristic message drops or toolset filtering, no hint
sniff stream cuts. tool_choice keeps its protocol meaning and the
tool-format contract still renders where the template needs it.
- MTPLX_AGENT_REWRITES=on restores the full legacy machinery (the test
suite pins it via the legacy_rewrites fixture).
- MTPLX_AGENT_REWRITES=off is a hard passthrough guarantee that beats
per-feature opt-ins and resolves tool prompts template-native (a
backend-required mode still wins: that is protocol, not policy).
- Per-feature MTPLX_*_COMPACT_THRESHOLD_CHARS env limits engage exactly
one compactor at the chosen limit, for harnesses that want a lower
cap than their own default.
- New --agent-rewrites {on,off} flag on serve and quickstart.
- Compaction wrapper text no longer names OpenCode on other clients.
- request_observability carries agent_rewrites; goldens regenerated
(single field drifted: request_pi_convergence_after_tools 14 -> 0).
…oussofal#282) Both Pi writers (app PiIntegration.sync and CLI mtplx/pi.py) replaced the whole providers.mtplx block on every launch, silently reverting user edits inside it: added vision input, a tuned thinkingLevelMap, explicit maxTokens, renamed models, extra headers (the intensifi report). New merge contract, identical in both writers: - MTPLX owns connection identity only: baseUrl, api, apiKey, authHeader and the x-mtplx-client header, because ports move between launches. - Every other key the user edited wins; MTPLX values fill gaps recursively. Model entries merge by id; user-added models and fields survive a sync untouched. - Stale MTPLX-owned model entries (our own previous 'mtplx-'-prefixed ids) are pruned so switching models does not accumulate dead picker rows. - The request-policy extension now carries an ownership marker: replace its content (dropping the marker and the mtplx identifiers) and it is yours, MTPLX never rewrites it again. Managed copies keep receiving updates; templates stay byte-identical across both writers. Tests: Python round-trip preserving vision/thinking/cap/name edits plus extension ownership; Swift mirrors both (654/654 green).
# Conflicts: # apps/MTPLXApp/Sources/MTPLXAppCore/Onboarding/RuntimeSetupService.swift # mtplx/cli.py # mtplx/commands/forge.py # mtplx/compressed_tensors.py # mtplx/gdn_capture.py # mtplx/mtp_patch.py
…pers, drop unused Mapping import
The bench handler applies the profile env block in-process by design; the youssofal#285 test file exercised it without isolation, so performance-cold's MTPLX_DROP_EVENTS leaked into the suite and silenced context-copy block events two files later (caught by the first full-suite run of the merged 2.9.2 tree).
…t-level decision)
The app's serve launcher exported the May-era transcript-rewrite envs (tool-result 1200-char compactor, read-inspection line caps, force-answer after 12 tools) on every coding-agent launch. Explicit envs re-arm those compactors past the engine's MTPLX_AGENT_REWRITES passthrough default, so app-launched daemons kept rewriting Pi/OpenCode transcripts after the engine-side cleanup. Receipt: a live Pi session showed 133,794 chars of tool results compacted at the 1200 threshold.
The receipts page for the project's founding claims, mirroring mtplx.com/history/: first exact speculative sampling with the native MTP heads on a Mac (2026-04-27, commit-stamped), shipped as an installable runtime 2026-05-02, llama.cpp reaching MTP 05-16 and hybrid GDN 08-03, vllm-metal documenting the cache-vs-speculation gap 08-10, and the oMLX kernel attribution. Every entry carries a public commit, PR, or changelog reference so the claims stay checkable. README gains a History section pointing at both copies. Prose authored by the founder's editing pass (recovered from release-stash-20260817); facts verified against MEASUREMENTS.md, the changelog, and the upstream PRs before landing.
The release gate's app/CLI parity pair caught the other half of the app fix: mtplx start still exported the read-inspection battery (72/8/120 + force-answer-after-12) on the shared block and the Pi compaction five (threshold 1200 + line caps) on the Pi lane. Explicit envs re-arm those compactors past the engine's passthrough default, so CLI-launched agent sessions were still being rewritten. Both launchers now export none of it, and a new parity test pins the full nine-key contract on all three surfaces.
Version bump, changelog, and release notes for the 2.9.2 patch release: passthrough-by-default serving endpoints and app launcher (youssofal#282), greedy trio default-on below a 12288-token context fence (youssofal#313/youssofal#315/youssofal#318), forge MTP norm convention decided per tensor set (youssofal#301) with a double-shift load guard (youssofal#306), vision canonicalization and cache-restore fixes (youssofal#327/youssofal#296), batched repetition stop (youssofal#311), content-free request log (youssofal#326), symlinked zshrc installer fix (youssofal#292), honest bench flags (youssofal#285), exactness kernel fixes (youssofal#319/youssofal#320), community ports (PR youssofal#273, youssofal#281, youssofal#330), and experimental opt-in fusion/crossrow/headcal/marathon levers.
… windows)
Second float4 bank (rows 4..7) in the partials kernel; compiles out at
QL <= 4 so the shipping D3 verify shape is byte-identical. Dispatch gate
and kernel contract widen to 2..8; explicit max_q_len still narrows.
Exactness: fp32-reference parametrized q_len 2..8 x offsets {515, 2048,
2051} green (26/26). Depth 4's verify window (q_len 5) now rides the
split-KV fast path at >=8k ctx instead of silently falling to fused.
…d, exact by construction Port of the Y-PC 3090 Session-2 lever (step −12.9% there): draft from the top-N frequency-ranked lm_head rows; the target verifies the full vocab, so ratio acceptance + residual correction stay exact for any proposal q. Design decisions with receipts: - Side-stamped head (_mtplx_frspec_draft_head), swapped in ONLY around the device draft core's warm+trace window and restored in a finally: legacy draft paths keep dense draft_q arrays indexed by real token id (generation.py ~9999), so a global swap would IndexError them. - Every mapping site width-guards at trace time (row width == ids width): an mx.compile retrace resolving the restored full head degrades to exact full-vocab drafting, never a silent mismap. - core['vocab_size'] now carries the distribution DOMAIN (full vocab) when FR-Spec is engaged — SparseDistribution.to_dense() scatters by real token id and would overflow a pruned-width domain. - CPU unit receipt: pruned(x) == full(x)[ids] max|diff| = 0.0. Env (default-off): MTPLX_FRSPEC_DRAFT=1, MTPLX_FRSPEC_VOCAB=<json>, MTPLX_FRSPEC_N. The Y-PC ranked-ids artifact (runs/draft_vocab.json, coverage 0.99728 on real traces, ceiling cost ≤2.2% over 8 positions) loads unchanged — same tokenizer family.
…CKED_WIDE + exactness tests
…), route grouped 7-16; 47/47 exact
…width-guarded local->full remap at the draft convergence point
…he tool-parse fallback, repair failure can never 500 a servable turn Full-gate catch: extending the reasoning-only repair to first turns made it fire on unclosed tool-call markup (the thinking splitter classifies post-<think> markup as reasoning), hijacking turns the established unclosed_tool_call fallback owns — and a repair-path exception (e.g. a tokenizer without the expected surface) 500d a request whose first pass was perfectly servable. The repair now skips any first pass carrying tool-control markers, and the whole attempt is contained: failure stamps reasoning_completion_repair_error and returns the original result; stream cancellation still propagates. Full gate after: 5218 passed / 0 failed.
…al#397 adapted) + portable tier + auto default + empty-answer repair 98K cold prefill +34.8% (ABBA, fan-verified, die-temp-gated), peak 91.4->83.0 GB; 131K at 810.6 tok/s; 262K — the youssofal#393 wedge scenario — completes at 738.7 tok/s / 87.36 GB peak. Live product-path QA: default serve offers the full 262,144 window honestly (lane-aware youssofal#393 pricing), 51K cold prompt at 913.8 tok/s / correct answer / healthy acceptance, warm restore 1.71s TTFT. Full gate 5218/0. Receipts in research MEASUREMENTS.md + LOG.md 2026-08-29 21:05 PDT.
…rebuild on failure A user's app-owned venv (Application Support/MTPLX/runtime-venv) had a torn mlx install: core.cpython-314-darwin.so could not dlopen against its own libmlx (Symbol not found: ...scaled_dot_product_attention...). The daemon died before /health on every launch and REINSTALLING THE APP CANNOT FIX IT: the venv survives reinstall, `mtplx --version` never imports mlx so the version floor stays green, and the bundled-wheel fingerprint still matches, so installOrUpdate reused the corpse forever. Reported via Twitter with `mtplx report --summary` receipts (FAIL mlx.import, macOS 26.4.1 / py3.14). The bootstrapper now proves the venv can actually run before trusting it: - Import probe (`venv/bin/python -I -c "import mlx.core, mtplx"`) on first adoption of each wheel; verdict cached in runtime-import-health.sha256 inside the venv (healthy steady-state launches pay one stat()). - A daemon death before /health writes a recheck breadcrumb (MTPLXBackendStore catch paths, classifier excludes cancellations and port conflicts), forcing a re-probe on the next launch — late breakage (foreign pip session, torn upgrade) self-heals in one relaunch. - A failed probe rebuilds the venv from scratch (venv --clear + full reinstall from the bundled wheel) with a "Repairing MTPLX runtime" status, and fresh installs must pass the probe before being vouched for. mtplx doctor's mlx.import fix text now states the real remedy (app relaunch self-repairs; CLI: pip --force-reinstall) instead of the no-op "pip install mlx". 9 new/updated Swift tests cover the marker fast path, first-adoption probe, breadcrumb override, --clear rebuild, and the death classifier.
…G14 device gates for 1024-thread kernels Issue youssofal#400 (M2 Max 96GB, Flash-Next Optimized Speed) reported two boot blockers this commit fixes; the third finding (committed-history verify stall) is the decode-cliff campaign's subject. 1) Memory guard refused the pack by exactly its own margin: flat 6 GiB floor margin + flat 16 GiB system reserve were sized on the 128GB receipt machine (weights 77.3 + 6 + 16 = 99.3 > 96), while the reporter demonstrably serves the pack healthy there with ~16 GiB unwired. Both terms now scale below their receipt machines and are UNCHANGED at 128GB+: system reserve = clamp(RAM/8, 8, 16) GiB (96GB -> 12); floor margin = 6 GiB at >=112GB RAM else clamp(RAM/32, >=2) GiB (96GB -> 3). 96GB now admits at floor 80.3 + reserve 12 = 92.3 <= 96 with the wired cap raised to the floor and 15.7 GiB left unwired — in line with the issue's own healthy receipt (wired 79.3 / free 16.7). Oversized packs still refuse; Laguna 64GB refusal and all existing cap tests unchanged. The refusal message now prints the actual scaled reserve. 2) Three turbo-armed kernels hard-crashed serve boot on G14 GPUs: hyper_connection_v3 (R1+R2), gdn_conv_norm, and its verify-rows variant dispatch structurally-1024-thread pipelines, and on M2/M3 register pressure caps the compiled pipelines at 896 threads — MLX raises at encode time, and pack contracts arm the envs, so the boot dies until the operator hand-exports three kill switches. Each kernel module now carries a one-shot device probe (dispatches the REAL pipeline on dummy family-shaped inputs — the limit is per-pipeline register pressure, so nothing cheaper proves it) and the qwen4_exp applies-gates consult it: unsupported devices land on the eager chain automatically with one log line. Probes verified green on M5 Max (no false negatives on healthy hardware); the rows parity test pre-warms the probe so its engagement counter stays exact.
Probe receipt (decodecliff battery, 2026-08-30, M5 Max, Flash-Next 91K warm turn): a 384-token completion read decode_tok_s=21.3 while its own sliding decode windows ran 50-62 tok/s — 10.7 s of session-restore machinery (snapshot-view COW divergence; prompt_state_unattributed_time_s) was charged INTO decode_elapsed_s because the rate split only subtracted measured prefill compute + cache-restore fetch. This is a big slice of the founder-visible "decode collapses to 20-25 tok/s at 80-90K in OpenCode": the engine decoded at ~50, the receipts (and every UI reading them) blamed decode for restore stalls. _generation_rate_fields gains non_decode_extra_s; the mtpk loop passes the full prompt-state span remainder + pre-first-token setup + the bank commit span, and the AR loop now measures its prompt-state span and does the same (stamping prompt_state_total/unattributed for AR receipts too). end_to_end_tok_s still tells the whole-wall truth. Unit tests replay the turn-4 receipt (21.3 -> 52.2 tok/s) and pin the clamp + default-zero semantics.
…t time The lazy put (kvcache-v2) keeps response tails flat by snapshotting the live cache as zero-copy views — but every unevaluated view holds a reference to a live cache buffer, blocking MLX buffer donation. The next turn's first write to each buffer then pays a full COW divergence copy. Historically the views only settled when the coalesced SSD cold-encode job got an idle window; under back-to-back agent turns (OpenCode) that window never arrives, and the divergence lands inside the following request: measured 10.7 s of prompt_state time on a warm 91K turn (decodecliff probe 2026-08-30), and the founder's session shows the same class (12.7 s TTFT on a warm 86K turn, 30 s postcommit timeout followed by a 44K re-prefill). put() now dispatches a settle job (per-array mx.eval over the entry's snapshot views) to the model-owner idle lane AHEAD of the SSD encode, newest-wins coalesced per session, off-switch MTPLX_SESSION_SNAPSHOT_SETTLE=0. Per-array granularity bounds any foreground request that lands mid-settle to ~one array (~10 ms). First receipts: the two battery arms that ran with settle live show zero stall events across six warm 91K turns (TTFT <= 2.2 s) vs an 11.1 s landmine in the pre-settle arm; dedicated on/off arms are running.
…wner-copy rewrite; decode-lane engagement counters Two corrections from tonight's battery receipts (MEASUREMENTS.md 2026-08-30): 1. Settle v1's per-array mx.eval was a physical no-op: evaluating a full-range lazy view ALIASES the source buffer (measured: zero allocation; mx.contiguous also no-ops on contiguous inputs), so the donation-blocking reference survived and the COW stalls reproduced with settle on. Rewritten to metal_copy_leaf owner copies with field-at-a-time entry rebinding (every intermediate state value-correct for concurrent restores; rebind + value equality pinned by test). 2. Even mechanically correct, the phase-3 A/B (on/off/off/on, warm 91K turns) falsified settle as a DEFAULT: stall magnitude is dominated by idle-lane/SSD scheduling nondeterminism (the next request queues behind multi-GB encode/settle jobs) and the added copy produced the worst observed stall (27.6 s). Default flipped OFF; the mechanism stays as an opt-in instrument with the falsification documented at the switch. The structural fix for the stall class is youssofal#391-style fixed-capacity QSA banks + restore-priority idle-lane preemption (LOG.md same date). Also: decode-lane engagement counters (decode_flash_skip / decode_gather / decode_dense_mask) on the QSA S=1 selector arms — the flash/gatherdec 91K arms read flat and future lane A/Bs need engagement receipts, per the counters law — and the decode-cliff probe driver (scripts/decodecliff_probe_20260830.py: process-per-arm serves, fan-verified, die-temp gated, salted prompts, receipt extraction).
…on-truth split The old assertion pinned decode_elapsed == elapsed - prompt_eval, i.e. the exact accounting 586d087 fixed (restore machinery silently counted as decode). Now asserts the full identity including cache_restore and prompt_state_unattributed, plus the new AR prompt-state stats. Also adds the pulse-cell driver script for instrumented decode captures. Full gate after fix: rc=0, zero FAILED.
Patch release over 2.10.0: the lightning sparse prefill lane (PR youssofal#397 by maceip, adapted; 98k prefill -35% time, 262k un-wedged at 87.4 GB where 2.10.0 hit 119 GB with zero tokens, youssofal#393), native Flash-Next image serving (M-RoPE, youssofal#328), 96 GB admission + M2/M3 threadgroup crash fixes (youssofal#400), tools-declared empty-answer repair, greedy Turbo exactness on the 27B, app runtime venv self-heal, and honest decode attribution in receipts. Stamps: pyproject / version.py / CITATION / uv.lock 2.10.1.
…s — ci job red since 2.10.0 on 7GB runners The seat tests pinned mtplx.memory_plan.detect_total_ram_bytes but the plan also consumes the host through _detect_total_ram_bytes_for_metal_caps, whose 75%-of-RAM memory limit rides into the plan as usable_bytes_override. On 7 GB GitHub macos-14 runners that collapsed the override to ~5 GiB: the 128G test came back machine-bound at 4096 and the 48G seats drifted. This exact no-mlx-smoke job was already red on the v2.10.0 release commit (run 33252270428) — pre-existing, not a 2.10.1 regression. The harness now routes the caps detector through the same per-test pinned source and clears MTPLX_MEMORY_LIMIT_BYTES/MTPLX_WIRED_LIMIT_BYTES. CI file set locally: 883 passed / 0 failed (was 3 failed on the runner).
… MTP throttling for 546k+ decode speed - Route S=1 QSA indexer scoring through multi-threadgroup grid parallelism (qsa_indexer_prefill_scores + Metal top-k radix kernel) past 16k tokens, removing single-threadgroup serial bottlenecks - Add quantized pooled key mirror (q8/q4) support to QSACache to cut indexer memory bandwidth by up to 87.5% - Implement adaptive MTP history window throttling capping draft history at 16k tokens when prompt depth > 262k - Add benchmark receipts validating 100% NIAH passkey recall and sustained 26.5 - 58.3 tok/s decode across 16k - 546k context depths
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
Author
|
if you use this can you merge one of my commits so I show up as a contributor? i know you give credit in the releases but im jobless and maybe that will help. ╮(╯_╰)╭ |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 582c22892b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ding, and diagnostics - Pass prompt length to adaptive MTP window calculation - Pad pooled keys buffer before group quantization and slice dequantized prefix - Avoid gathering quantized rows twice in rows-gather attention lane - Ensure clean nonresident SSD streaming without host synchronizations - Report per-request QSA engagement counts from snapshot deltas - Clamp explicit server context window to model capability - Calibrate benchmark haystack generation for exact token depth targets
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 226a83281b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…remental pooled quant, and memory plan - Preserve sparse access when dequantizing QSA KV in flash and rows-gather lanes - Quantize only newly changed pooled groups incrementally on cache append - Apply engine RAM fraction bounds when Metal limit exists in memory planner - Preserve profile ceiling for compiled verify max context - Restore rows-gather ceiling to 8 to keep copy-block widths dense at 16k floor
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc8e7be5f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ty snapshot restore, MTP adaptive restore, and n-gram rows - Avoid repeated O(T) dequantization during decode by caching and incrementally updating the transposed FP32 pooled view in QSACache - Reset underlying KV cache to pristine state when restoring empty QSACache snapshots - Rebuild or trim restored MTP history when adaptive window requirements change across prompts - Wire MTPLX_NGRAM_HOT_ROWS into sidecar sizing
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09dda4fadd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… for last_window - Inspect the restored MTP cache offset rather than calculating an inferred window from prefix length - Trim excess history tokens when restoring a full committed entry for a last_window request - Check true stored history capacity in candidate restore matching
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e850b649d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… telemetry, snapshot quant validation, and deep cap clamping - Route compiled S=1 decode at deep context (T >= MTPLX_QSA_PARALLEL_INDEXER_MIN_CONTEXT) to multi-threadgroup scorer - Count compiled QSA routes in request stats counters (decode_flash_skip, gather_rows, decode_dense_mask) - Reject incompatible dense vs quantized KV restores in QSACache.state setter and include QSA quantization in policy fingerprint - Clamp MTPLX_MTP_HISTORY_DEEP_CAP to at least 1
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 372f63ae8b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tation and update cache nbytes - Derives QSACache.pooled_f32_view solely from pooled_quant_t when pooled_bits is active to keep scoring bit-consistent across cold, restored, and compiled states - Stops retaining unnecessary unquantized pooled_f32_t mirror under quantized pooled mode - Accurately counts both pooled_quant_t and pooled_f32_t allocations in QSACache.nbytes
Contributor
Author
lv_0_20260831211212_github-10mb-1.mp4 |
Owner
|
Update: handled on our side instead, nothing for you to configure. Main's history was rewritten tonight so your commits carry your account linked identity, and the 2.10.2 ports of #395 and #396 now carry you as author outright, four commits total. The contributors graph will show you as GitHub reindexes. Adopted work lands with the contributor's linked identity from now on. Note the rewrite changed hashes, so open branches need a rebase onto current main. |
youssofal
force-pushed
the
main
branch
2 times, most recently
from
September 1, 2026 08:07
2382dfd to
8bc4d88
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores fast decode throughput and bounds verification latency across deep context prefill rungs (16k up to 546k tokens) on Qwen4-Exp (
Qwen3.8-Flash-Next-MTPLX-Bare-Speed):qsa_indexer_prefill_scores) and Metal 2048-bin radix top-k (qsa_indexer_prefill_topk_metal), eliminating single-threadgroup serial scan bottlenecks across all 40 GPU cores.q8/q4):QSACachein quantized format whenMTPLX_QSA_POOLED_BITS/MTPLX_QSA_KV_BITSis enabled, reducing DRAM bandwidth and memory footprint by up to 87.5% (MTPLX_MTP_HISTORY_DEEP_CAP), keeping speculative verification rounds bounded belowBenchmark Receipts (M5 Max 128GB, Bare-Speed Model)
Validated across 7 context rungs on
Qwen3.8-Flash-Next-MTPLX-Bare-Speedon clean 2.10.1:Test Plan
5223 passed, 28 skipped in 187swith 0 failures.