llama.cpp-rdna-lab is a hardware-focused fork of
ggml-org/llama.cpp for local AI on
Windows with two AMD RDNA4 GPUs. It combines a maintained subset of the
llama.cpp runtime with a local web GUI (GUI 2.0, in gui2/), reproducible
benchmark/autotune tooling, long-context work, and AMD-specific Vulkan and
ROCm/HIP optimizations.
The desktop application is branded RDNA LLM Studio. The repository keeps
upstream llama-* executable names and source structure so that runtime
compatibility and upstream synchronization remain straightforward.
The primary workload is agentic coding with Qwen3.8-27B: large cold prompts, single-user requests, long contexts, tool use, vision, and speculative decode. The main performance priority is prompt evaluation. MTP is kept only when its decode gain does not impose an unacceptable prefill cost.
This is a specialized research and production fork, not a drop-in replacement for every upstream platform. Results and defaults are tuned for the reference dual-RX 9070 XT machine described below.
- README — this file: overview, quick start, and build guide
- Contributing — contribution rules and workflow
- License & Security — MIT license, with private vulnerability reporting and secure-use guidance in SECURITY.md
- Fork Details — fork-only features, backend fixes, and recommended runtime profiles
- Performance — current benchmark tables and matched lane contracts
- Benchmarking — canonical benchmark methodology and history
- Supported backends — backend policy for this fork
| Area | Current focus |
|---|---|
| Host platform | Windows 11 on AMD AM4 |
| Accelerators | 2x Radeon RX 9070 XT 16 GB (gfx1201) |
| Backends | ROCm/HIP, Vulkan, and CPU |
| Primary model | Qwen3.8-27B Q4_K_M with MTP |
| Experimental model | Ternary Bonsai 27B PQ2_0 on CPU and ROCm |
| Main objective | Maximum cold prompt evaluation without sacrificing useful decode speed |
| Serving | OpenAI-compatible llama-server plus the local web GUI (GUI 2.0) |
The fork is currently substantially faster than the measured stock upstream checkout on the same long-prompt contract. See Fork vs Stock Upstream for the exact matched runs.
Production and daily work stay on master; larger experiments run on named
branches and are merged back selectively. Snapshot: 2026-08-26.
| Branch | Status | Latest state |
|---|---|---|
master |
Stable baseline | D131 R9 MTP window audit PASS, multi-stream scale-view fix (276121b7e). 2026-08-26: merged RPC stack + qwen4exp port (1d1aa5a6c). |
rpc-vulkan |
Merged, kept for continuation | RPC backend restored for Vulkan offload; merged into master 2026-08-26. Recovers the RPC backend (removed upstream), fixed quantized alloc_size OOB crash (q3_K/q6_K), split ggml-rpc into a layered file stack, and reached the 3-GPU 12K RPC lane target 1314 ptps / 23.6 t/s (target ≥1277 ptps; PPL 4.0148 ≈ baseline), up from 839 ptps after alloc-size caching, a 16 MB send buffer, and -ts 0.9,0.6,1.5. See RPC resume playbook. |
dflash2 |
Paused, to be resumed | DFlash2 block-diffusion drafter port for Qwen3.8-27B (local 2-tap depthwise conv + candidate selector, Qwen3.8-27B-DFlash2-Q4_K_M.gguf in models/), 5 commits ahead of master. Paused at a measured ROCm checkpoint (2026-08-21): opt-in multi-scheduler reuse gives 29.59 aggregate / 36.90 decode / 785.58 prompt tok/s on 498+128 vs 29.00/36.33/749.70 for the single-cache control, with the strict n=3 boundary/parity gate bit-exact. Open items include the upstream batched-greedy divergence and the Vulkan long-decode crash. Resume: ROCm playbook, Vulkan playbook. |
research/vulkan-decode |
Backlog, to be resumed | D104 (Q6_K prefill dispatch) and D105 (decode bandwidth) closed; Q4_K16 port log reached Vulkan PPL parity with bf16 (6.6124 vs 6.6202 on 256 chunks); MTP n=3 measured decode optimum 1.83x. 15 commits ahead, 10 behind master (rebase needed before continuation). |
research/vulkan-fp8-kv |
Backlog, to be resumed | D131 R9: fp8 K per-block scale (LLAMA_VK_F8_K_SCALE), K-scale broadcast fix, MTP window audit PASS, multi-stream scale-view fix. C2 closed: f8_direct prefill lost 23.5% to preconvert. 20 commits ahead, 10 behind master. |
- Documents
- At a Glance
- Active Branches
- Project Goals
- Supported Backends and Models
- Remote GPU (RPC) Backend
- Reference System
- Performance Summary
- Fork Highlights
- Quick Start
- Build Requirements
- MTP Behavior
- Vision
- Benchmarking
- Repository Layout
- Development
- License and Security
- Maximize Qwen3.8 prompt-evaluation throughput for agent workloads.
- Use both GPUs without moving the active working set into system RAM.
- Make MTP improve decode while keeping long-prompt prefill close to baseline.
- Provide a practical GUI for building, launching, monitoring, and autotuning.
- Keep performance claims reproducible through cold, lane-locked benchmarks.
- Keep the fork maintainable by carrying only the backends and upstream changes that are useful on this machine.
Current non-goals include broad accelerator portability and native support for NVIDIA CUDA, Metal, SYCL, OpenCL, CANN, or other removed upstream backends.
| Backend | Role | Status |
|---|---|---|
| ROCm/HIP | Primary prompt-eval, long-context MTP, and RDNA4 runtime | Supported and preferred for prompt-heavy MTP work |
| Vulkan | General AMD runtime and backend comparison | Supported; competitive for decode-heavy work; q8/MTP path fixed (D094) |
| CPU | Fallback, conversion, sanity checks, and tests | Supported |
ROCm still builds HIP-compatible kernels from ggml/src/ggml-cuda. That is an
internal HIP implementation detail and does not mean that this fork supports
NVIDIA hardware. See Supported Backends.
| Model / feature | CPU | ROCm/HIP | Vulkan | Notes |
|---|---|---|---|---|
| Qwen3.8 GGUF (Q4_K_M primary; Qwen3.6 family also supported) | Yes | Yes | Yes | Primary supported family |
| Qwen3.8 NextN MTP | Yes | Yes | Yes | Requires an MTP-enabled GGUF |
Ternary Bonsai 27B PQ2_0 |
Yes | Yes | Not yet | Native loader, CPU kernels, and HIP MMQ/MMVQ path |
| Qwen3.5/3.6/3.8 vision projector | Yes | Yes | Yes | Use a matching mmproj-*.gguf |
| DFlash | Research | Research | Research | Not a recommended production profile |
D094 (2026-08-07, tested on Qwen3.6-27B-Q4_K_M.gguf and
Qwen3.6-27B-Q3_K_S_mtp.gguf, 2x RX 9070 XT): the Vulkan q8_0 vec/mmq
numerical divergence vs ROCm was root-caused and fixed (CUDA-style dp4a
accumulation, round-half-away q8_1 quantize, mmq variant-B math). MTP
acceptance recovered from 0.33 to 0.80+ (52k-token drafts; target 0.53) and
the f16-KV 49K lane now beats ROCm
(prompt 1719.92 vs 1679.20 tok/s, decode 43.10 vs 32.33, aggregate 6.1358 vs
5.7655 TPS). See BENCHMARKS.md and
Q4_K_M_RESULTS.md.
Qwen3.8-27B-Q4_K_M is the primary practical Qwen model on this 2x16 GB
machine (rebased 2026-08-14; it shares the qwen35 architecture family with
Qwen3.6 and runs the same MTP/vision paths). The one-copy ROCm scheduler and
bounded Q8 Flash Attention route make its measured 49K and 98K lanes viable.
Q3_K_S (Qwen3.6) remains the secondary choice for maximum
context/VRAM headroom, vision, and Q3-specific kernel research. PQ2_0 is an
experimental Prism format and should not be confused with conventional Q2_0
quantization.
The fork restores the upstream-removed RPC backend to offload a slice of the model to a remote Vulkan GPU over TCP. The lab setup uses a local RTX 3080 (10 GB) as the third device alongside the two RX 9070 XT cards.
- Code layout — the RPC stack lives in
ggml/src/ggml-rpc/(rpc_types.hprotocol,rpc_common.cpptransport/queue/hash,rpc_client.cpp,rpc_server.cpp,transport.*); the standalone server binary istools/rpc/rpc-server. - Protocol — version
5.0.1with a fail-closed check against old servers. The client and server must be built from the same tree. - Key optimizations — alloc-size caching (removes hundreds of round-trips
per ubatch), F16 activation staging at the boundary, async outbound queue and
run-ahead, opt-in block-Q8_0 activation wire (
GGML_RPC_ACT_Q8_0), and the MTPgraph_recomputehash fix. - Run —
llama-server --rpc <host>:<port> -dev Vulkan1,Vulkan0,RPC0 -sm layer -ts <f1>,<f2>,<f3>withRPC0naming the remote device.
- 12K 3-GPU lane (target reached 2026-08-23) — 1314 ptps / 23.6 t/s,
PPL 4.0148 ≈ baseline, up from 839 ptps after alloc-size caching, a 16 MB
send buffer, and
-ts 0.9,0.6,1.5(layers 20/13/32; the 3080 holds 8.9/10 GB). The target was ≥1277 ptps, i.e. 80% of the solo 1596.8 ptps. - MTP through RPC (fixed) —
graph_recomputehash mismatch made the verify graph read another context's weights (acceptance 1.4%). After the fix: acceptance 59.7% (89/149), decode 37.0 t/s (12K, n=4, loopback; local control 40.15 t/s). - 94K lane —
1328.65/25.12on RPC vs local1351.18/25.05(−0.3% over the wire). - Opt-in Q8_0 activation wire — 94K
1063.87/39.47, +4.63% prefill vs F16 control1016.82/37.59, wire traffic 10.49 → 5.57 MiB per ubatch (−46.9%). Still gated behind the quality/PPL check. - Caveat — the 27B numbers above were captured before the causal-mask (non-causal contour) fix on the RPC path; a re-measure is pending before they are treated as final.
The stack was first validated on a small 9B model as a development harness: F16 activation staging gave +20% prefill, loopback overhead is about −10% prompt / −12% decode, and over 1 GbE LAN the 3080 loses −38% / −61% — the network, not the GPU, is the bottleneck at this scale.
- Practice — run the server from SYSTEM (
schtasks) so it survives WinRM sessions, keep the NV shader cache warm, stop gracefully (/exit, never hard-kill a server with a loaded model), and use live logs (tee).
Detailed design and iteration notes: docs/research/rpc-vulkan/RPC_ARCHITECTURE.md, RPC_CHANGES.md, RPC_BASELINE_TABLE.md, and RPC_PREFILL_RESUME_PLAYBOOK.md.
- Windows 11
- AMD Ryzen 7 5800X3D, 8 cores / 16 threads
- 64 GB system RAM
- 2x AMD Radeon RX 9070 XT, 16 GB VRAM each, RDNA4
gfx1201 - AMD ROCm/HIP SDK 7.1 for Windows
- AMD proprietary Vulkan driver
- Main model:
Qwen3.8-27B-Q4_K_M.gguf - Vision projector:
mmproj-F16.gguf
The two GPUs are normally used with layer split, not tensor split. GPU1 is the preferred output device because GPU0 also drives the desktop. Device order is backend- and workload-sensitive even with two identical cards, and the best Vulkan order is not identical for every lane. Exact routes are recorded with each benchmark instead of being presented as a universal default. PCIe topology, driver version, background GPU load, KV type, and model residency can materially change the numbers below.
Full tables, lane contracts, and evidence links live in PERFORMANCE.md. Headline snapshot (2026-08-14, Qwen3.8-27B-Q4_K_M rebaseline; FlashAttention, cold prompts, no reuse):
| Backend | Lane | Mode | Prompt TPS | Decode TPS | Aggregate TPS | Acceptance |
|---|---|---|---|---|---|---|
| Vulkan | 49,152 | q8_0 none | 1532.79 | 26.05 | 5.1016 | - |
| Vulkan | 49,152 | q8_0 MTP n2 | 1637.44 | 48.56 | 5.9465 | 81.8% |
| ROCm | 49,152 | f8_e4m3 native none | 1713.67 | 22.20 | 8.6528 | - |
| ROCm | 49,152 | f8_e4m3 native MTP n2 | 1716.79 | 39.96 | 6.0332 | 78.2% |
On Qwen3.8 the Vulkan FP8 prompt advantage over q8 is roughly parity at
12K/49K/98K; ROCm native FP8 still holds +4.0% prompt, +3.1% decode and
+3.7% aggregate at 49K. The 98K Vulkan last-12-f16 MTP profile is
context-research material (60.8% acceptance vs q8's 71.5%), not a default
recommendation.
Same-day, same-session A/B (bench2, GPU-free precheck, warmup on, seed 42,
-c 98304 -b 8192 -ub 1024 -ngl 999 --flash-attn on --cache-type-k/v q8_0 -dev Vulkan1,Vulkan0 -sm layer -ts 1,1 -fit off, synthetic prompt). Fork =
build-vulkan-gcc16 (GCC 16.2, 572cdc0f7 + MTP load gating), stock =
ggml-org/llama.cpp b10666 (GCC 16.2).
Spec none, L0-L3 r1:
| Lane | Fork prefill | Stock prefill | Δ% | Fork decode | Stock decode | Δ% |
|---|---|---|---|---|---|---|
| L0 (8K) | 1158.6 | 827.1 | +40.1% | 29.47 | 30.22 | −2.5% |
| L1 (16K) | 1411.0 | 1141.1 | +23.7% | 29.04 | 29.66 | −2.1% |
| L2 (49K) | 1353.5 | 1148.1 | +17.9% | 27.75 | 27.91 | −0.6% |
| L3 (98K) | 1172.4 | 976.8 | +20.0% | 26.45 | 25.89 | +2.2% |
Spec draft-mtp (n=2), L0-L3 r1:
| Lane | Fork prefill | Stock prefill | Δ% | Fork decode | Stock decode | Δ% |
|---|---|---|---|---|---|---|
| L0 (8K) | 1034.0 | 898.1 | +15.1% | 39.28 | 49.89 | −21.3% |
| L1 (16K) | 1346.4 | 1097.2 | +22.7% | 52.72 | 51.33 | +2.7% |
| L2 (49K) | 1412.4 | 1015.8 | +39.1% | 39.43 | 48.85 | −19.3% |
| L3 (98K) | 1241.3 | 853.4 | +45.5% | 34.65 | 41.14 | −15.8% |
Fork wins prompt eval everywhere; decode is at parity for none, but stock
keeps the MTP-decode edge on L0/L2/L3 (L1 is fork +2.7%). Artifacts:
vk-l0123-20260829-{1133,1135,1414,1340}.
How it was run (short commands):
# fork (default build-vulkan/bin)
python scripts/bench2.py run --level 0,1,2,3 --runs 1 --backend vk
python scripts/bench2.py run --level 0,1,2,3 --runs 1 --backend vk \
--server-extra "--spec-type draft-mtp --spec-draft-n-max 2"
# stock: point build-vulkan/bin at the stock binary first, then the same commands
cp bench2-bins/stock_vk/llama-server.exe build-vulkan/bin/llama-server.exePipeline parallelism (PP) note (2026-08-29, deferred): on Vulkan, enabling
PP lowers prompt prefill (fork L0-MTP: 1034.0 -> 792.4 ptps) but restores
decode (stock L0-MTP: 40.11 -> 49.98 tps with PP on; stock PP-off -ngl 65
was 691.3 / 40.11). The fork PP path is not yet equivalent to stock, so the
PP-for-MTP reconciliation (keep prefill advantage, catch the stock decode) is
left for further investigation.
The fork's main differences from stock, beyond benchmark tooling and the GUI:
- Dual-GPU control — explicit device order, layer split, and
LLAMA_OUTPUT_DEVICEplacement instead of relying on automatic defaults. - Qwen MTP on both backends — backend-resident NextN handoff, ROCm KV-only sparse history, warm Vulkan verification topology.
- AMD kernel work — RDNA4 Q3_K/PQ2_0 HIP kernels, rocWMMA FlashAttention, native E4M3 FP8 KV routes, Vulkan q8/mmq correctness fixes.
- Reproducible benchmarking —
scripts/agent_workload_bench.pywith canonical history files and lane contracts.
See Fork Details for the complete feature list, diagnostics and rollback controls.
Install Python GUI dependencies and launch the application from the repository root:
python -m pip install -r gui/requirements-gui.txt
python run.pyrun.bat and start-gui.bat are also available. In the GUI:
- Open Build & Setup and configure Vulkan, ROCm/HIP, or CPU.
- Build
llama-serveror select an existing compatible build. - Open Launch Server and select a local GGUF model.
- Start with
Spec: Noneto establish a baseline. - For an MTP-enabled GGUF, select MTP and use depth 3 as the current general Vulkan/ROCm starting point.
- For vision, enable the projector and select
models/mmproj-F16.gguf. - In Benchmark / Autotune, use the recommended explicit device order for
reproducible dual-GPU tests.
Autoremains useful for discovery, but it is not a stable benchmark contract. - Validate batch, ubatch, KV, split, and spec settings at the intended context length. Short-prompt winners do not automatically remain best at 49K.
Model files are not part of the source tree history. Put local GGUF files in
models/ or select them from another local directory.
The reference builds are Windows x64 builds. A clean machine needs:
- Git and 64-bit Python 3.11 or newer with
pip; - CMake 3.14 or newer and Ninja (tested with CMake 3.29 and Ninja 1.12);
- Visual Studio Build Tools 2022 with Desktop development with C++, the MSVC v143 toolset, and a Windows 10 or 11 SDK;
- the current AMD display driver, including the Vulkan runtime;
- full LunarG Vulkan SDK with
glslc,spirv-as,spirv-dis, andspirv-valfor Vulkan/FP8 shader builds; - AMD ROCm/HIP SDK 7.1 for Windows for ROCm builds;
- Strawberry Perl for Windows ROCm configuration and the reference MinGW Vulkan toolchain;
- OpenSSL development files. HTTPS is enabled by default; use
-DLLAMA_OPENSSL=OFFonly when HTTPS/model downloads are not required.
The tested Vulkan build uses the GCC 13.2 MinGW-w64 toolchain bundled with
Strawberry Perl. A MinGW executable also needs libgcc_s_seh-1.dll,
libstdc++-6.dll, and libwinpthread-1.dll either beside the executable or on
PATH. The GUI launch environment handles the configured toolchain; for a
manual launch, put C:\Strawberry\c\bin before other MinGW installations on
PATH to avoid loading incompatible runtime DLLs.
The tested ROCm build uses clang.exe and clang++.exe from HIP SDK 7.1, not
MSVC as the compiler, but still links against MSVC v143 and Windows SDK host
libraries. Strawberry Perl is also required. A full HIP compilation is memory
intensive; 64 GB RAM and -j 4 are recommended for this fork. Allow roughly
30 GB of free disk space for source, two build trees, and one local model.
Install the Python side and verify the native tools before opening the GUI:
python -m pip install --upgrade pip
python -m pip install -r gui/requirements-gui.txt
cmake --version
ninja --version
glslc --version
spirv-as --version
spirv-dis --version
spirv-val --versionThe GUI's Build & Setup tab checks the configured dependencies and creates backend-specific build directories. Manual equivalents are shown below.
cmake -S . -B build-cpu -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build-cpu -j 4 --target llama-server$env:VULKAN_SDK = "C:\VulkanSDK\<version>"
$env:PATH = "$env:VULKAN_SDK\Bin;C:\Strawberry\c\bin;$env:PATH"
cmake -S . -B build-vulkan -G Ninja `
-DGGML_VULKAN=ON `
-DCMAKE_C_COMPILER=C:\Strawberry\c\bin\gcc.exe `
-DCMAKE_CXX_COMPILER=C:\Strawberry\c\bin\g++.exe `
-DCMAKE_BUILD_TYPE=Release
cmake --build build-vulkan -j 4 --target llama-server$env:HIP_PATH = "C:\Program Files\AMD\ROCm\7.1"
$env:ROCM_PATH = $env:HIP_PATH
$env:CMAKE_PREFIX_PATH = "$env:HIP_PATH\lib\cmake"
$env:PATH = "$env:HIP_PATH\bin;C:\Strawberry\perl\bin;C:\Strawberry\c\bin;$env:PATH"
cmake -S . -B build-rocm -G Ninja `
-DGGML_HIP=ON `
-DAMDGPU_TARGETS=gfx1201 `
-DGGML_HIP_MMQ_MFMA=ON `
-DGGML_HIP_ROCWMMA_FATTN=ON `
-DGGML_HIP_NO_VMM=ON `
-DGGML_OPENMP=OFF `
-DCMAKE_C_COMPILER="$env:HIP_PATH\bin\clang.exe" `
-DCMAKE_CXX_COMPILER="$env:HIP_PATH\bin\clang++.exe" `
-DCMAKE_BUILD_TYPE=Release
cmake --build build-rocm -j 4 --target llama-serverROCm uses clang from the HIP SDK but still needs the Windows SDK and MSVC host
libraries. Missing kernel32.lib, msvcrtd.lib, or similar files indicates an
incomplete Build Tools environment. See the full Build Guide.
The fork includes rocWMMA 7.1 headers under third_party/rocwmma; no separate
rocWMMA SDK install is required for the command above.
MTP accelerates token generation; it does not make the target model's prompt prefill free. ROCm uses selected long-range KV blocks plus the recent prompt tail and keeps NextN hidden states on their backend, avoiding a complete draft prefill and the previous GPU-to-RAM-to-GPU round trip. Vulkan uses a host handoff by default because keeping unmasked NextN output resident over the whole Vulkan prompt was substantially slower.
Practical rules:
- Use at least 128 output tokens when benchmarking MTP. Very short runs are dominated by the first target-verification graph.
- Compare MTP and
nonewith the same model, prompt, output length, KV type, batch/ubatch, device split, and background load. - Depth 3 is the current robust starting point. Higher depth is not automatically faster because acceptance falls and verification batches grow.
- For prompt-dominated requests with short answers,
nonecan still win wall time even when MTP decode is much faster. - Non-zero Windows Shared memory is not by itself proof that MTP is reading KV from RAM. Check for a throughput cliff and compare process Dedicated/Shared; the current 72K lane adds only about 62 MiB Shared during MTP prefill.
- Set
LLAMA_MTP_DEVICE_HANDOFF=0only as a diagnostic rollback to the old host hidden-state path.
Qwen3.6 vision requires a projector that matches the text model architecture
and embedding dimension. In the GUI, enable Vision and select
models/mmproj-F16.gguf. The equivalent server argument is:
--mmproj models/mmproj-F16.gguf
Use Spec: None for the first image request so vision-pipeline issues can be
separated from speculative decoding.
The canonical runner starts an isolated OpenAI-compatible server, injects a real repository snapshot, records prompt/decode timings, and updates the live history files:
python scripts/agent_workload_bench.py --helpImportant history files:
build_logs/agent-workload/BENCH_RUNS.csvbuild_logs/agent-workload/BENCH_RECENT.mdbuild_logs/agent-workload/BENCH_LANES.mddocs/research/RESULTS_LOG.md
Performance work should use neighboring controls. Background GPU applications,
driver power state, warm shader caches, prompt-cache reuse, or a different
output length can otherwise create a false improvement. Record an explicit
-dev route for every dual-GPU result; the GUI now defaults new ROCm and Vulkan
benchmark configurations to the measured recommended order instead of Auto.
| Path | Purpose |
|---|---|
gui2/ |
GUI 2.0: the local web UI (FastHTML + HTMX) |
src/, common/, include/ |
llama runtime and speculative pipeline |
ggml/src/ggml-vulkan/ |
Vulkan backend and generated shaders |
ggml/src/ggml-hip/ |
ROCm/HIP build integration |
ggml/src/ggml-cuda/ |
Shared HIP-compatible kernel implementation |
ggml/src/ggml-cpu/ |
CPU backend |
scripts/agent_workload_bench.py |
Benchmark and autotune runner |
PERFORMANCE.md |
Current benchmark tables and lane contracts |
FORK_DETAILS.md |
Fork-only features, fixes, and runtime profiles |
docs/research/ |
Accepted, rejected, and diagnostic performance work |
docs/vulkan/ |
Vulkan architecture and validation rules |
Read AGENTS.md before changing the fork. Upstream changes are ported selectively according to UPSTREAM_SYNC.md; removed backends are not restored automatically during synchronization.
When reporting performance, include the model, backend, device order, split, context, actual prompt tokens, output tokens, batch/ubatch, KV types, speculative mode, cache policy, and background load. A faster isolated number is useful only when its lane and tradeoffs are visible.
The runtime is derived from ggml-org/llama.cpp.
Upstream changes are reviewed and ported selectively so they do not silently
restore removed backends or invalidate AMD-specific behavior. This repository
is distributed under the MIT License; bundled third-party components
retain their own notices and licenses.
Security issues are handled privately. See SECURITY.md for the reporting policy, covered topics (runtime, ggml, and GGUF tooling), and secure-use guidance for untrusted models, inputs, and networks. Do not report vulnerabilities as public issues before the disclosure window closes.