command-center/agent/managed-skills/llamacpp-vram-accounting/SKILL.md

name, description

name	description
llamacpp-vram-accounting	Produce and interpret a per-device VRAM/memory breakdown for a llama.cpp or halogen LLM deployment on the Dank fleet: we/weights vs KV vs compute pool, discrete vs UMA accounting, and the measurement traps that make the numbers lie.

llama.cpp / halogen VRAM accounting (Dank fleet)

Use when asked where the memory went for an LLM service on Dankclaw/Dankserver: "what's the VRAM breakdown", "why does it not fit", "how much headroom is left for the co-tenants", or before sizing a card for a new model.

0. Decide which kind of memory you are talking about

device	kind	read it with
RTX 3090 (Dankclaw)	discrete GDDR6X	nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader
gfx1151 iGPU (Dankclaw)	UMA — "VRAM" is host RAM	/sys/class/drm/card1/device/mem_info_gtt_{total,used} (real, 114 GiB) vs mem_info_vram_{total,used} (2 GiB carve-out window)
R9700s (Dankserver)	discrete	rocm-smi figure is authoritative (64 GiB claimed, SVM-backed)

Per-process truth on NVIDIA: nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader.

Never add the iGPU's GTT to host RAM as if they were separate pools, and never quote a UMA engine's total as "VRAM": halogens's 95.4 GiB is 68 GiB of pinned host pages (mlock + page cache) plus ~33.7 GiB inside the device budget. free -g is the arbiter for what the co-tenants have left.

1. Get the engine's own ledger first

podman logs frank-large 2>&1 | grep -aiE "weights pinned|kv pool|left for everything|overlay|vision|budget"

Halogen prints the numbers that matter: weights pinned (and how many contiguous 2 MiB blocks remain), the device budget (host RAM − resident weights − reserve), the KV pool size and pool-vs-working split, and the final "left for everything else" block count. Contiguous-block count is the honest fragmentation signal — 240 blocks / 12.3 GiB after a reboot vs 475 / 14.7 GiB before means the same model did not come back into the same hole.

2. For llama.cpp: the print sites exist, but no artifact here has them

• src/llama-model.cpp:1827 — per-backend model buffer size (unconditional LLAMA_LOG_INFO)
• src/llama-context.cpp:692 — compute buffer size
• plus a KV self-size line at context creation

Traps verified on 2026-09-16:

• llama-bench suppresses the loader's INFO — its logs contain only the device banner and the results table, so a bench matrix leaves no memory accounting behind.
• llama-server under systemd did not put them in the journal either — sudo journalctl -u frank-small | grep -c "model buffer size" returned 0 for the unit's entire history. Check with grep -c before claiming a log has the breakdown.
• Recipe that works: launch the model once through llama-cli (or llama-server) and keep the first ~200 lines; add -v if the level filters them.

3. Otherwise compose it — arithmetic on measured tensor sizes

Read the GGUF's tensor table once (names + shapes are enough) and group by role, e.g. Flash-Next (111 GiB):

group	size	per layer
ffn_{gate,up,down}_exps	75.0 GiB	1.5625 GiB (experts travel with their layer)
per_layer_token_embd (PLE)	32.8 GiB	one table
everything else (attn/GDN/etc)	3.2 GiB	0.067 GiB

And KV ≈ 2 KiB per position per f16 attention layer (12 attention layers on Flash-Next => ~24 KiB/position; halogen's ledger says ~28 KiB/position including block scratch). Multiply by the derived context (see §5).

Then take the measured per-device total as the constraint and let the remainder be compute/graph scratch. Vendor delta is real and sizeable: same 7 layers + KV on the 3090 came to 12.80 GiB under Vulkan and 14.99 GiB under CUDA — ~3.3 GiB vs ~1.1 GiB of graph/compute pool.

4. Interpretation traps

• mmap PLE is reclaimable; -ot makes it not. With everything on the iGPU the 32.8 GiB PLE sits in page cache (reclaimable). In a layer split, where -ot is disallowed, it becomes a pinned device buffer in GTT. "Move layers to the card" therefore converts reclaimable cache into pinned device memory and does not free room for co-tenants.
• A served model's footprint is not its weight size. frank-small: 22,582–22,592 MiB of 24,576 for an 11.29 GiB model — weights + f16 KV at 131,072 × 1 slot + BF16 projector + CUDA compute pool.
• "Usable" is total minus driver/container overhead (~24.1 GiB on a 24,576 MiB card).
• Do the capacity subtraction before designing: 12 attention layers × 1.62 GiB = 19.4 GiB, plus 200k f16 KV (9.2 GiB) = 28.6 GiB > 24.1 — which is why a KV-on-VRAM design on this pair is only feasible to ~64k context.

5. Context lengths: derive, never remember

llama-bench never logs the context; it computes (tools/llama-bench/llama-bench.cpp:1293):

cparams.n_ctx = n_prompt + n_gen + n_depth;

Defaults: n_prompt 512, n_gen 128, n_depth 0, n_batch 2048, n_ubatch 512, type_k/type_v F16, flash_attn AUTO, split_mode LAYER, reps 5.

So -p 512,16384 -n 64 = contexts 576 / 16,448 / 64, and -n 32 -d 65536 = 66,080 / 65,568. A tg row runs with n_prompt = 0, which is why a plain tg number is ~zero depth: -d is the only thing that puts KV in the picture. Cross-check by reading the table's own test column (pp512 @ d65536, tg32 @ d65536).

6. Reporting shape that survives review

Label every number: measured (nvidia-smi / sysfs / engine ledger line), derived (arithmetic on tensor sizes + derived context), or inference (e.g. depth extrapolation). State the context each memory figure was taken at, and whether KV was resident — 12.8 GiB at 65,568 tokens is not the same claim as 12.8 GiB at 200k.