server : expose per-device memory usage on /metrics and GET /memory

[tobocop2]

Overview

Closes #26129

llama-server now reports its memory usage: how many bytes it allocated on each GPU (and in host RAM), split into model weights, context/KV cache, and compute buffers - plus each device's total and free memory.

Two surfaces over the same data: /metrics for Prometheus/Grafana, and GET /memory for JSON consumers.

--metrics enables the Prometheus surface; a new --memory flag enables the JSON one (the --slots pattern: one flag per endpoint).

The server has always known these numbers - it prints them at startup and shutdown - but never exposed them anywhere a program can read. I run several llama-server instances sharing a set of GPUs and need to know what each one actually allocated on each card. Today the only way is scraping log lines, which breaks whenever the log format changes.

1. common: made the per-device summary usable on its own.

The numbers were computed inside the function that prints the shutdown table, so the only way to get them was as text.

They now come from common_memory_breakdown_get(): one row per device, plus one for host memory. The print function just renders those rows - same table as before. --fit's memory projection had its own copy of this aggregation and now uses the same function.

One accounting fix along the way: the context's output buffer was missing from the breakdown (and so from the printed table). It's counted now.

No new use of the staging header src/llama-ext.h.

2. server: the breakdown on /metrics (Prometheus) and GET /memory (JSON), plus the model's layer count and the multimodal projector's memory.

llamacpp:memory_model_bytes{device="MTL0"} 83349984
llamacpp:memory_model_bytes{device="Host"} 13191648
llamacpp:memory_total_bytes{device="MTL0"} 22906503168
llamacpp:model_n_layer 12

{
  "n_layer": 12,
  "data": [
    {
      "name": "MTL0",
      "model": 83349984,
      "context": 0,
      "compute": 22559744,
      "total": 22906503168,
      "free": 22800187392
    },
    {
      "name": "Host",
      "model": 13191648,
      "context": 500640,
      "compute": 3684352
    }
  ]
}

The device label / name is the backend's own device name - the same CUDA0 / MTL0 names that --list-devices prints and --device accepts.

/metrics stays behind --metrics. /memory gets its own --memory flag, advertised on /props and refusing with 501 when disabled, like the other optional endpoints.

The JSON response is an object so fields can be added later without breaking clients - n_layer next to data is the first.

Both endpoints reuse the existing internal metrics task - no new task type. /slots uses that task too and skips the device query.

With a multimodal projector loaded, rows carry its per-device memory too - via a new mtmd_get_ctx_memory_usage() that reads the live clip contexts instead of re-estimating from the file. Real output, tinygemma3 on Metal:

{
  "n_layer": 8,
  "data": [
    {"name": "MTL0", "model": 40707072, "context": 510656512, "compute": 165136416, "mmproj": 2181120, "total": 22906503168, "free": 22187376640},
    {"name": "Host", "model": 35660288, "context": 4195328, "compute": 152064032, "mmproj": 12288}
  ]
}

3. Bug fix the new metrics ran into: /metrics rounded large values.

The exporter formatted every value as a double, which silently rounds anything past 6 significant digits. Byte counts made it obvious:

# before
llamacpp:memory_total_bytes{device="MTL0"} 2.29065e+10

# after
llamacpp:memory_total_bytes{device="MTL0"} 22906503168

This was already quietly affecting prompt_tokens_total past a million tokens. Integers are now emitted exactly; float metrics (the throughput averages) keep their old formatting. # HELP / # TYPE are also emitted once per metric name now that one name can carry several {device=...} series.

Additional information

Using it

Grafana / Prometheus, straight off /metrics - graph VRAM per card, or alert on headroom:

llamacpp:memory_model_bytes{device="CUDA0"}

# alert: less than 10% of the device left free
(llamacpp:memory_free_bytes / llamacpp:memory_total_bytes) < 0.10

Python, off GET /memory:

import requests

resp = requests.get("http://localhost:8080/memory").json()
print("layers:", resp["n_layer"])
for row in resp["data"]:
    print(row["name"], f'{row["model"] / 2**20:.1f} MiB weights')

layers: 12
MTL0 79.5 MiB weights
Host 12.6 MiB weights

C++, from anything that links common:

for (const common_memory_breakdown_row & row : common_memory_breakdown_get(ctx)) {
    printf("%s: model %zu, context %zu, compute %zu bytes\n",
        row.name.c_str(), row.mem.model, row.mem.context, row.mem.compute);
}

Verification

Check	macOS M1 Pro (Metal)	macOS (CPU-only)	RunPod A40 (CUDA)
Build: llama-server, llama-cli, llama-perplexity, llama-bench, llama-fit-params	ok	ok	llama-server
/metrics vs the shutdown breakdown table, same run	exact match (table below)	-	exact match (table below)
GET /memory JSON	ok (output above)	-	ok (below)
tools/server/tests/unit/test_metrics.py (9 tests, incl. a vision model exercising mmproj)	pass	-	-
ASan/UBSan under concurrent /metrics + /memory + embeddings, and again with tinygemma3 (mmproj path)	0 reports	-	-
TSan under concurrent /metrics + /memory, embeddings, and tinygemma3 (mmproj path)	-	0 warnings	-
/metrics wakes a sleeping server (--sleep-idle-seconds)	ok	-	-

Metal cross-check (nomic-embed-text-v1.5.Q4_K_M):

Field	Printed table	/metrics
MTL0 model	79 MiB	79.49 MiB
MTL0 compute	21	21.51
MTL0 total	21845	21845.34
MTL0 free	21743	21743.95
Host model	12	12.58
Host compute	3	3.51

RunPod A40 (CUDA 12.4) cross-check, same model:

Field	Printed table	/metrics
CUDA0 model	66 MiB	66.92 MiB
CUDA0 compute	21	21.51
CUDA0 total	45498	45498.00
CUDA0 free	45109	45109.62

GET /memory on the pod (captured one revision earlier - with the output-buffer accounting, Host.context additionally reports the ~0.5 MiB output buffer, as in the Metal example above):

{
  "n_layer": 12,
  "data": [
    {
      "name": "CUDA0",
      "model": 70170624,
      "context": 0,
      "compute": 22559744,
      "total": 47708110848,
      "free": 47301066752
    },
    {
      "name": "Host",
      "model": 13191648,
      "context": 0,
      "compute": 3684352
    }
  ]
}

A CPU-only run on the same pod (GPU masked) returns the host-plus-buffer-type shape (Host and CPU_REPACK rows, no total/free).

Prebuilt Binaries

Prebuilt binaries of this PR for all platforms (built with the repo's own release workflow on my fork): https://github.com/tobocop2/llama.cpp/releases/tag/build-memory-metrics-release-b10139-b11c3a9

Notes

• n_layer rides the memory surfaces because layers are the unit of memory planning (-ngl).
• Only device names are exposed, not descriptions - the name is what --device takes. Label values are still escaped per the exposition format, since RPC device names embed the user's endpoint string.

Related demand: #23926 (users asking for these numbers after the load report moved out of the default log level).

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - I designed this change and understand all of it. I used AI assistance to write much of the code and to check it: sanitizer runs (ASan/UBSan/TSan), thread-safety analysis of the metrics task path, and review passes for allocations and project conventions. I reviewed the result line by line.

[ggml-gh-bot]

Hi @tobocop2, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[tobocop2]

Hi @tobocop2, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

* **PR Template not respected**: Please respect the [template](https://github.com/ggml-org/llama.cpp/blob/master/.github/pull_request_template.md?plain=1) when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Updated