serverAppConfigs/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

---
type: reference
title: Llama Cpp Serving Sizing
description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
timestamp: 2026-10-05T00:00:00-07:00
---

# Llama Cpp Serving Sizing

## 1. Slot-pool sizing against a headroom band

Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
keeping a requested amount of VRAM free — from measurement, never arithmetic
guesswork.

1. **Fix the invariants.** Read the payload's KV geometry (attention layers
   only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
   `n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
   `kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
   state buffer — slot count is a memory term, not free.
2. **Two boots on the real card** (production stopped, per the bench law),
   each near the target; record `nvidia-smi` used/total plus the boot log's
   `n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
   `non-KV = used − (tokens × KiB/token)`; the two rows separate
   weights/buffers/graphs from the new demand (a 918 MB projector bills
   ~1.4 GiB).
3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
   KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
   slot gets an equal integer window. Boot and measure; if outside the band,
   walk one step — never re-derive from a formula.
4. **Propagate the per-slot number — it is the client contract:**
   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
   or long conversations die with `400 exceed_context_size_error` instead of
   compacting; compaction/summary budgets sized to the old window get
   re-checked; endpoint docs and the rollback table move too.
5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
   concurrent requests answered in parallel with N+1 queued, and — if a
   projector ships — a solid-color image probe through an **`image_url`
   content part** expecting the exact color. The legacy top-level `images`
   field is silently ignored, so a probe that uses it measures a
   hallucination, not vision.

Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
measured rows in the service `.env` comment block and mirror the non-KV price
in the bats budget gate so a shape that busts the band fails the suite instead
of the boot.

## 2. Per-device VRAM accounting ("where did the memory go?")

First decide which kind of memory is in question:

| device | kind | read it with |
|---|---|---|
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
| R9700s (Dankserver) | discrete | `rocm-smi --showmeminfo vram` → 34,208,743,424 B = 32,624 MiB (31.86 GiB) per card, read 2026-10-18 — **not** the 64 GiB "SVM-backed" figure this row once carried ([[Dankserver System Specs]]) |

Per-process truth on NVIDIA:
`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
co-tenants.

- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
  contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
  after reboot vs 475/14.7 GiB before means the model did not come back into
  the same hole), device budget, KV pool/working split, "left for everything
  else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
  and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
  `master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
  ("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
  ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
  re-read 2026-10-18) — but **`llama-bench`
  suppresses loader INFO and a systemd unit did not put them in the journal**
  (verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
  keeping the first ~200 lines.
- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
  role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
  table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
  layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
  GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
  graph/compute pool).
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
  "move layers to the card" converts reclaimable cache into device memory and
  frees nothing for co-tenants. A served model's footprint ≠ weight size
(a retired frank-zealot shape: 22.6 of 24.1 GiB for an 11.29 GiB model; the card
today reads 24,576 MiB = 24.0 GiB total, `nvidia-smi --query-gpu=memory.total`,
2026-10-18, and the deployed payload is the PQ2_0 pair
[[frank-zealot-llama-prism-bonsai2-pq20]]). Do the capacity
  subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
  28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
  `n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
  flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
  576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
  puts KV in the picture.
- **Label every number** measured / derived / inference, and state the context
  each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
  as at 200k.

## 3. Prompt cache and state sizing

`--cache-ram` (alias `-cram`, `common/arg.cpp:1719`) is HOST RAM for saved
prompt states — unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (alias
`-ctxcp`/`--swa-checkpoints`, `common/arg.cpp:1701`) is a different mechanism:
per-slot KV snapshots; `0` disables them. Both flags are current upstream; the
two defaults quoted here (8192 MiB and 32) are the era's values and were not
re-readable in the upstream sources checked *(unverified 2026-10-18)* — print
them from the binary (`llama-server --help`) before sizing on them.

**State law** (compute before trusting any cap): a state is the serialized KV
of the whole sequence + constant recurrent state + the draft context's KV:
`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
`prompt state size` lines — the two must agree.

**The journal is the instrument:** `exceeds cache size limit … skipping` =
cap binding in production; `making room … removing oldest` / `cache size
limit reached` = eviction, and the printed size proves the active cap. Do NOT
claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
repeats that counter is 0 by construction.

**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
cell stores only the last position), so with `--ctx-checkpoints 0` there is no
checkpoint to fall back on and the server logs `forcing full prompt
re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
intra-slot reuse only survives when the request echoes the assistant's
previous reply verbatim (a pure user resend is always a full re-prefill even
at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
conversation state ACROSS slots, so a cap that refuses deep states costs
minutes of re-prefill whenever slots are reassigned; state restoration only
runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
calls `prompt_load`.

**Verify a change:** live argv carries the flag (not the deploy log); `/health`
200 + real completion; VRAM unchanged and host RAM mostly free; no new
`exceeds cache size limit` lines, with a `making room` entry larger than the
old cap as direct proof. If the ceiling itself was not exercised live, say so —
crossing it needs a >120k-token conversation and must not run beside a live
service (bench law).

## 4. QSA flash-attn assert fix (qwen4exp)

`--flash-attn on` aborts graph build for qwen4exp models:
`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
`ggml_fill` requires F32.

Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
only at the attention call when `flash_attn` is enabled
(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
`scripts/patches/qwen4exp-qsa-f32-mask.patch`.

Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
it; CIRU, upstream and stock builds do not. Validate live: load with
`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
in the timings. Context cap for full-model residency on that shape stayed
32,768.

## Related

- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot-llama-prism-bonsai2-pq20]])
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side