Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
78 lines
5.7 KiB
Markdown
78 lines
5.7 KiB
Markdown
---
|
||
name: llamacpp-vram-accounting
|
||
description: "Produce and interpret a per-device VRAM/memory breakdown for a llama.cpp or halogen LLM deployment on the Dank fleet: we/weights vs KV vs compute pool, discrete vs UMA accounting, and the measurement traps that make the numbers lie."
|
||
---
|
||
|
||
# llama.cpp / halogen VRAM accounting (Dank fleet)
|
||
|
||
Use when asked *where the memory went* for an LLM service on Dankclaw/Dankserver: "what's the VRAM breakdown", "why does it not fit", "how much headroom is left for the co-tenants", or before sizing a card for a new model.
|
||
|
||
## 0. Decide which kind of memory you are talking about
|
||
|
||
| device | kind | read it with |
|
||
|---|---|---|
|
||
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv,noheader` |
|
||
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (real, 114 GiB) vs `mem_info_vram_{total,used}` (2 GiB carve-out window) |
|
||
| R9700s (Dankserver) | discrete | rocm-smi figure is authoritative (64 GiB claimed, SVM-backed) |
|
||
|
||
Per-process truth on NVIDIA: `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader`.
|
||
|
||
**Never** add the iGPU's GTT to host RAM as if they were separate pools, and never quote a UMA engine's total as "VRAM": halogens's 95.4 GiB is 68 GiB of **pinned host pages** (mlock + page cache) plus ~33.7 GiB inside the device budget. `free -g` is the arbiter for what the co-tenants have left.
|
||
|
||
## 1. Get the engine's own ledger first
|
||
|
||
```bash
|
||
podman logs frank-large 2>&1 | grep -aiE "weights pinned|kv pool|left for everything|overlay|vision|budget"
|
||
```
|
||
|
||
Halogen prints the numbers that matter: weights pinned (and how many contiguous 2 MiB blocks remain), the device budget (`host RAM − resident weights − reserve`), the KV pool size and pool-vs-working split, and the final "left for everything else" block count. Contiguous-block count is the honest fragmentation signal — 240 blocks / 12.3 GiB after a reboot vs 475 / 14.7 GiB before means the same model did not come back into the same hole.
|
||
|
||
## 2. For llama.cpp: the print sites exist, but no artifact here has them
|
||
|
||
- `src/llama-model.cpp:1827` — per-backend `model buffer size` (unconditional `LLAMA_LOG_INFO`)
|
||
- `src/llama-context.cpp:692` — `compute buffer size`
|
||
- plus a KV self-size line at context creation
|
||
|
||
Traps verified on 2026-09-16:
|
||
|
||
- **`llama-bench` suppresses the loader's INFO** — its logs contain only the device banner and the results table, so a bench matrix leaves *no* memory accounting behind.
|
||
- **`llama-server` under systemd did not put them in the journal either** — `sudo journalctl -u frank-small | grep -c "model buffer size"` returned **0** for the unit's entire history. Check with `grep -c` before claiming a log has the breakdown.
|
||
- Recipe that works: launch the model **once** through `llama-cli` (or `llama-server`) and keep the first ~200 lines; add `-v` if the level filters them.
|
||
|
||
## 3. Otherwise compose it — arithmetic on measured tensor sizes
|
||
|
||
Read the GGUF's tensor table once (names + shapes are enough) and group by role, e.g. Flash-Next (111 GiB):
|
||
|
||
| group | size | per layer |
|
||
|---|---|---|
|
||
| `ffn_{gate,up,down}_exps` | 75.0 GiB | **1.5625 GiB** (experts travel with their layer) |
|
||
| `per_layer_token_embd` (PLE) | 32.8 GiB | one table |
|
||
| everything else (attn/GDN/etc) | 3.2 GiB | 0.067 GiB |
|
||
|
||
And KV ≈ 2 KiB per position per f16 attention layer (12 attention layers on Flash-Next => ~24 KiB/position; halogen's ledger says ~28 KiB/position including block scratch). Multiply by the *derived* context (see §5).
|
||
|
||
Then take the measured per-device total as the constraint and let the remainder be compute/graph scratch. Vendor delta is real and sizeable: same 7 layers + KV on the 3090 came to **12.80 GiB under Vulkan** and **14.99 GiB under CUDA** — ~3.3 GiB vs ~1.1 GiB of graph/compute pool.
|
||
|
||
## 4. Interpretation traps
|
||
|
||
- **mmap PLE is reclaimable; `-ot` makes it not.** With everything on the iGPU the 32.8 GiB PLE sits in page cache (reclaimable). In a layer split, where `-ot` is disallowed, it becomes a pinned device buffer in GTT. "Move layers to the card" therefore converts reclaimable cache into pinned device memory and does not free room for co-tenants.
|
||
- **A served model's footprint is not its weight size.** frank-small: 22,582–22,592 MiB of 24,576 for an 11.29 GiB model — weights + f16 KV at 131,072 × 1 slot + BF16 projector + CUDA compute pool.
|
||
- **"Usable" is total minus driver/container overhead** (~24.1 GiB on a 24,576 MiB card).
|
||
- Do the capacity subtraction *before* designing: 12 attention layers × 1.62 GiB = 19.4 GiB, plus 200k f16 KV (9.2 GiB) = 28.6 GiB > 24.1 — which is why a KV-on-VRAM design on this pair is only feasible to ~64k context.
|
||
|
||
## 5. Context lengths: derive, never remember
|
||
|
||
`llama-bench` never logs the context; it computes (`tools/llama-bench/llama-bench.cpp:1293`):
|
||
|
||
```c
|
||
cparams.n_ctx = n_prompt + n_gen + n_depth;
|
||
```
|
||
|
||
Defaults: `n_prompt 512`, `n_gen 128`, `n_depth 0`, `n_batch 2048`, `n_ubatch 512`, `type_k`/`type_v` **F16**, `flash_attn AUTO`, `split_mode LAYER`, `reps 5`.
|
||
|
||
So `-p 512,16384 -n 64` = contexts **576 / 16,448 / 64**, and `-n 32 -d 65536` = **66,080 / 65,568**. A `tg` row runs with `n_prompt = 0`, which is why a plain `tg` number is ~zero depth: `-d` is the only thing that puts KV in the picture. Cross-check by reading the table's own test column (`pp512 @ d65536`, `tg32 @ d65536`).
|
||
|
||
## 6. Reporting shape that survives review
|
||
|
||
Label every number: **measured** (nvidia-smi / sysfs / engine ledger line), **derived** (arithmetic on tensor sizes + derived context), or **inference** (e.g. depth extrapolation). State the context each memory figure was taken at, and whether KV was resident — 12.8 GiB at 65,568 tokens is not the same claim as 12.8 GiB at 200k.
|