serverAppConfigs/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Llama Cpp Serving Sizing	Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.	llama.cpp vram kv-cache slots flash-attn sizing	2026-10-05T00:00:00-07:00

Llama Cpp Serving Sizing

1. Slot-pool sizing against a headroom band

Goal: choose CTX_SIZE/PARALLEL/KV types so N conversations fit a GPU while keeping a requested amount of VRAM free — from measurement, never arithmetic guesswork.

1. Fix the invariants. Read the payload's KV geometry (attention layers only, KV heads, head_dim, KV dtype bytes) and the per-slot rule n_ctx_slot = CTX_SIZE / PARALLEL. A hybrid model with kv_unified = 'false' gives each slot its own cache AND a fixed per-slot state buffer — slot count is a memory term, not free.
2. Two boots on the real card (production stopped, per the bench law), each near the target; record nvidia-smi used/total plus the boot log's n_slots/n_ctx_slot: shape A = current/cheap, shape B = new demand. non-KV = used − (tokens × KiB/token); the two rows separate weights/buffers/graphs from the new demand (a 918 MB projector bills ~1.4 GiB).
3. Solve the band: KV_tokens ≤ (card − non-KV − headroom_low) / KiB/token, round the pool DOWN to a multiple of PARALLEL × 4096 so every slot gets an equal integer window. Boot and measure; if outside the band, walk one step — never re-derive from a formula.
4. Propagate the per-slot number — it is the client contract: models.yml contextWindow = n_ctx_slot (NOT the model's native window), or long conversations die with 400 exceed_context_size_error instead of compacting; compaction/summary budgets sized to the old window get re-checked; endpoint docs and the rollback table move too.
5. Prove behaviour: boot-log slot count, nvidia-smi free in band, N concurrent requests answered in parallel with N+1 queued, and — if a projector ships — a solid-color image probe through an image_url content part expecting the exact color. The legacy top-level images field is silently ignored, so a probe that uses it measures a hallucination, not vision.

Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is allocated whole at boot so post-boot memory.used IS the ceiling; record both measured rows in the service .env comment block and mirror the non-KV price in the bats budget gate so a shape that busts the band fails the suite instead of the boot.

2. Per-device VRAM accounting ("where did the memory go?")

First decide which kind of memory is in question:

device	kind	read it with
RTX 3090 (Dankclaw)	discrete GDDR6X	nvidia-smi --query-gpu=memory.total,memory.used,memory.free
gfx1151 iGPU (Dankclaw)	UMA — "VRAM" is host RAM	/sys/class/drm/card1/device/mem_info_gtt_{total,used} (114 GiB) vs mem_info_vram_* (2 GiB carve-out)
R9700s (Dankserver)	discrete	rocm-smi figure (64 GiB, SVM-backed)

Per-process truth on NVIDIA: nvidia-smi --query-compute-apps=pid,process_name,used_memory. Never add the iGPU's GTT to host RAM as a separate pool; free -g is the arbiter for co-tenants.

• Engine ledger first. Halogen prints weights pinned (+ remaining 2 MiB contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB after reboot vs 475/14.7 GiB before means the model did not come back into the same hole), device budget, KV pool/working split, "left for everything else". llama.cpp prints at llama-model.cpp:1827 (per-backend model buffer) and llama-context.cpp:692 (compute buffer) — but llama-bench suppresses loader INFO and a systemd unit did not put them in the journal (verified grep -c = 0). Recipe: one launch through llama-cli/llama-server keeping the first ~200 lines.
• Otherwise compose from tensor sizes. Group the GGUF tensor table by role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention layer, × derived context. Vendor delta is real: same layers+KV cost 12.80 GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of graph/compute pool).
• Traps: mmap'd PLE is reclaimable page cache; -ot makes it pinned — "move layers to the card" converts reclaimable cache into device memory and frees nothing for co-tenants. A served model's footprint ≠ weight size (frank-small: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV = 28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
• Context lengths: derive, never remember. llama-bench computes n_ctx = n_prompt + n_gen + n_depth (defaults 512/128/0, type_k/v F16, flash_attn AUTO, split_mode LAYER). -p 512,16384 -n 64 = contexts 576/16,448; a plain tg row has ~zero depth — -d is the only thing that puts KV in the picture.
• Label every number measured / derived / inference, and state the context each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim as at 200k.

3. Prompt cache and state sizing

--cache-ram (MiB, default 8192) is HOST RAM for saved prompt states — unpinned, on demand, moves no VRAM. --ctx-checkpoints (default 32) is a different mechanism: per-slot KV snapshots; 0 disables them.

State law (compute before trusting any cap): a state is the serialized KV of the whole sequence + constant recurrent state + the draft context's KV: per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) + draft_per_token. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft): 16×4×256×2×2 = 64 KiB/token + 4 KiB draft = 68 KiB/token; a full 196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's prompt state size lines — the two must agree.

The journal is the instrument: exceeds cache size limit … skipping = cap binding in production; making room … removing oldest / cache size limit reached = eviction, and the printed size proves the active cap. Do NOT claim a raise is proven by prompt_tokens_cached_total rising — on exact repeats that counter is 0 by construction.

Hybrid + zero-checkpoint trap: for a hybrid (DeltaNet/SSM + attention) model, seq_pos_min sits at the END of the slot's own prompt (a recurrent cell stores only the last position), so with --ctx-checkpoints 0 there is no checkpoint to fall back on and the server logs forcing full prompt re-processing … (likely due to SWA or hybrid/recurrent memory). Consequences: intra-slot reuse only survives when the request echoes the assistant's previous reply verbatim (a pure user resend is always a full re-prefill even at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a conversation state ACROSS slots, so a cap that refuses deep states costs minutes of re-prefill whenever slots are reassigned; state restoration only runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never calls prompt_load.

Verify a change: live argv carries the flag (not the deploy log); /health 200 + real completion; VRAM unchanged and host RAM mostly free; no new exceeds cache size limit lines, with a making room entry larger than the old cap as direct proof. If the ceiling itself was not exercised live, say so — crossing it needs a >120k-token conversation and must not run beside a live service (bench law).

4. QSA flash-attn assert fix (qwen4exp)

--flash-attn on aborts graph build for qwen4exp models: GGML_ASSERT(a->type == GGML_TYPE_F32) at ggml.c:5371 through build_attn_qsa. Root cause: build_attn_qsa calls ggml_fill on the F16-cast kq mask returned by get_kq_mask() when flash attn is on, but ggml_fill requires F32.

Fix (mirrors the MLA sparse path in llm_graph_context::build_attn): run the top-k mask surgery on the F32 base mask (inp->self_kq_mask), cast to F16 only at the attention call when flash_attn is enabled (kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k, GGML_TYPE_F16) : kq_mask_top_k). The patch lives at scripts/patches/qwen4exp-qsa-f32-mask.patch.

Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes it; CIRU, upstream and stock builds do not. Validate live: load with --flash-attn on, health 200, a real completion finishing "stop", journal clean of GGML_ASSERT/cudaMalloc/missing tensor, and flash-attn engaged in the timings. Context cap for full-model residency on that shape stayed 32,768.

Related

• GGUF Compatibility — what a payload is before it is priced
• frank-small-llama-mimo-q6k-2x200704 — the current two-boot priced shape
• frank-medium-llamacpp-dflash2 — where the 68 KiB/token datum was measured
• DFlash DSpark Speculative Decoding — the spec A/B protocol on llama.cpp
• Dual-R9700-Cooperation / x4-pair-experiment — the multi-device side