feat(observability): llama.cpp metrics PodMonitor and pinned contract (#1186)

[joryirving]

What

Adds the producer half of the inference tier:

• config/monitoring/llamacpp-podmonitor.yaml — scrapes llama.cpp /metrics from every InferenceService the operator creates.
• docs/observability/llamacpp-metrics.md — the pinned metric contract, in the same shape as amd-gpu-metrics.md.

Why

Fixes #1186

Slice 2 of #700. Per the discussion on #1186, two of the three metric names pinned there do not exist in llama.cpp's /metrics, so #1187 would build panel queries against metrics that never arrive:

pinned in #1186	reality
llamacpp:requests_processing	exists
llamacpp:tokens_per_second	no such metric
llamacpp:kv_cache_usage_ratio	no such metric — nothing matching kv_cache is exposed

How

Scraping. The PodMonitor selects on the operator's own inference.llmkube.dev/service label rather than naming servers, so a new InferenceService is picked up with no monitoring change. Each series is relabelled with service, model, runtime and namespace, so #1187's panel can break down per model without hardcoding a target.

The static scrape_configs in prometheus-scrape-config.yaml are left as-is. They remain the right shape for an external Prometheus that cannot do service discovery in this cluster; the PodMonitor is the default for a cluster running prometheus-operator.

Contract. Throughput is pinned as two metrics rather than one — llamacpp:predicted_tokens_seconds (decode) and llamacpp:prompt_tokens_seconds (prefill). They bottleneck on different resources and degrade independently, so a combined figure hides one collapsing while the other holds. I watched a server fall from 56 to 5 tokens/s of decode with prefill unchanged; an averaged number would have read as a mild dip rather than a 10x regression.

llamacpp:requests_deferred is added as the queueing signal. It rises only once every parallel slot is occupied, which is what separates "busy" from "oversubscribed" — better than inferring queueing from latency.

KV-cache occupancy is documented as a known gap rather than substituted. llamacpp:n_busy_slots_per_decode is a reasonable saturation proxy but measures slot occupancy, not cache residency, and renaming it would put the same drift back into the contract that pinning it was meant to prevent.

Scope. Deliberately does not touch charts/llmkube/dashboards/amd-gpu-observability.json — that is #1187, and it should build against this contract once pinned. The inference stanza in amd-gpu-metrics.md now points at the new contract and gains requests_deferred; its metrics-contract markers are preserved.

Assisted-by: Claude Code (drafted the manifest, the contract doc and this description; I verified every metric name, type and description against my own running servers, confirmed the PodMonitor live, and checked the kustomize render).

Checklist

• Tests added/updated — no Go code changes; this is a manifest plus documentation, so there is nothing for the Go suite to exercise. Verified instead by kubectl kustomize config/monitoring rendering the PodMonitor, and by reading every documented metric, type and help string from live servers' own metric metadata (a Strix Halo gfx1151 node plus four other InferenceServices, scraped through this PodMonitor for 15 days).
• make test passes locally — unaffected, no Go changes
• make lint passes locally — unaffected, no Go changes
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance disclosed above, per CONTRIBUTING.md
• Documentation updated

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!