Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
177 lines
10 KiB
Markdown
177 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
title: Llama Cpp Serving Sizing
|
||
description: "Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache."
|
||
tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Llama Cpp Serving Sizing
|
||
|
||
## 1. Slot-pool sizing against a headroom band
|
||
|
||
Goal: choose `CTX_SIZE`/`PARALLEL`/KV types so N conversations fit a GPU while
|
||
keeping a requested amount of VRAM free — from measurement, never arithmetic
|
||
guesswork.
|
||
|
||
1. **Fix the invariants.** Read the payload's KV geometry (attention layers
|
||
only, KV heads, head_dim, KV dtype bytes) and the per-slot rule
|
||
`n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
|
||
`kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
|
||
state buffer — slot count is a memory term, not free.
|
||
2. **Two boots on the real card** (production stopped, per the bench law),
|
||
each near the target; record `nvidia-smi` used/total plus the boot log's
|
||
`n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
|
||
`non-KV = used − (tokens × KiB/token)`; the two rows separate
|
||
weights/buffers/graphs from the new demand; a projector is billed after the
|
||
fit at roughly 1.5x its file size ([[frank-zealot]] Contract has that
|
||
measurement).
|
||
3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
|
||
KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
|
||
slot gets an equal integer window. Boot and measure; if outside the band,
|
||
walk one step — never re-derive from a formula.
|
||
4. **Propagate the per-slot number — it is the client contract:**
|
||
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
|
||
or long conversations die with `400 exceed_context_size_error` instead of
|
||
compacting; compaction/summary budgets sized to the old window get
|
||
re-checked; endpoint docs and the rollback table move too.
|
||
5. **Prove behaviour:** boot-log slot count, `nvidia-smi` free in band, N
|
||
concurrent requests answered in parallel with N+1 queued, and — if a
|
||
projector ships — a solid-color image probe through an **`image_url`
|
||
content part** expecting the exact color. The legacy top-level `images`
|
||
field is silently ignored, so a probe that uses it measures a
|
||
hallucination, not vision.
|
||
|
||
Gotchas: shorter slots can make decode FASTER (~5 % measured); the KV pool is
|
||
allocated whole at boot so post-boot `memory.used` IS the ceiling; record both
|
||
measured rows in the service `.env` comment block and mirror the non-KV price
|
||
in the bats budget gate so a shape that busts the band fails the suite instead
|
||
of the boot.
|
||
|
||
## 2. Per-device VRAM accounting ("where did the memory go?")
|
||
|
||
First decide which kind of memory is in question:
|
||
|
||
| device | kind | read it with |
|
||
|---|---|---|
|
||
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
|
||
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` vs `mem_info_vram_*`; the aperture and carve-out values are [[Dankclaw System Specs]]'s |
|
||
| R9700s (Dankserver) | discrete | `rocm-smi --showmeminfo vram`; the per-card byte/MiB value (and the stale "64 GiB SVM-backed" figure it replaced) is [[Dankserver System Specs]]'s |
|
||
|
||
Per-process truth on NVIDIA:
|
||
`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
|
||
iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
|
||
co-tenants.
|
||
|
||
- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
|
||
contiguous blocks — the honest fragmentation signal: a block census that drops
|
||
across a restart means the model did not come back into the same hole; the
|
||
before/after census this was learned on is that era's, on [[frank-dragoon]]),
|
||
device budget, KV pool/working split, "left for everything
|
||
else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
|
||
and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
|
||
`master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
|
||
("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
|
||
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
|
||
re-read 2026-10-18) — but **`llama-bench`
|
||
suppresses loader INFO and a systemd unit did not put them in the journal**
|
||
(verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
|
||
keeping the first ~200 lines.
|
||
- **Otherwise compose from tensor sizes.** Group the GGUF tensor table by
|
||
role (Flash-Next example: 75.0 GiB experts at 1.5625 GiB/layer, 32.8 GiB PLE
|
||
table, 3.2 GiB everything else), KV ≈ 2 KiB/position per f16 attention
|
||
layer, × derived context. Vendor delta is real: same layers+KV cost 12.80
|
||
GiB under Vulkan vs 14.99 GiB under CUDA (~3.3 vs ~1.1 GiB of
|
||
graph/compute pool).
|
||
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
|
||
"move layers to the card" converts reclaimable cache into device memory and
|
||
frees nothing for co-tenants. A served model's footprint is not its weight
|
||
size: the MiMo Q5_K_M era booted a 6.40 GiB payload to **20,530 MiB used** of the
|
||
3090's addressable 24,576 — the Eras row and its addressable ≠ advertised law are on
|
||
[[frank-zealot]]. What the card reports used is weights + KV + compute buffers +
|
||
projector, never weights alone. Do
|
||
the capacity subtraction before designing, on the card number the owner page
|
||
quotes, not the advertised one.
|
||
- **Context lengths: derive, never remember.** `llama-bench` computes
|
||
`n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
|
||
flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
|
||
576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
|
||
puts KV in the picture.
|
||
- **Label every number** measured / derived / inference, and state the context
|
||
each figure was taken at — 12.8 GiB at 65,568 tokens is not the same claim
|
||
as at 200k.
|
||
|
||
## 3. Prompt cache and state sizing
|
||
|
||
`--cache-ram` (alias `-cram`, `common/arg.cpp:1719`) is HOST RAM for saved
|
||
prompt states — unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (alias
|
||
`-ctxcp`/`--swa-checkpoints`, `common/arg.cpp:1701`) is a different mechanism:
|
||
per-slot KV snapshots; `0` disables them. Both flags are current upstream; the
|
||
two defaults quoted here (8192 MiB and 32) are the era's values and were not
|
||
re-readable in the upstream sources checked *(unverified 2026-10-18)* — print
|
||
them from the binary (`llama-server --help`) before sizing on them.
|
||
|
||
**State law** (compute before trusting any cap): a state is the serialized KV
|
||
of the whole sequence + constant recurrent state + the draft context's KV:
|
||
`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
|
||
draft_per_token`. The worked qwen35-hybrid instance of that formula (full layer
|
||
count, f16 KV, MTP draft) and the slot-size table it prices are on
|
||
[[frank-medium-llamacpp-dflash2]]; cross-check any such arithmetic against the
|
||
journal's `prompt state size` lines — the two must agree.
|
||
|
||
**The journal is the instrument:** `exceeds cache size limit … skipping` =
|
||
cap binding in production; `making room … removing oldest` / `cache size
|
||
limit reached` = eviction, and the printed size proves the active cap. Do NOT
|
||
claim a raise is proven by `prompt_tokens_cached_total` rising — on exact
|
||
repeats that counter is 0 by construction.
|
||
|
||
**Hybrid + zero-checkpoint trap:** for a hybrid (DeltaNet/SSM + attention)
|
||
model, `seq_pos_min` sits at the END of the slot's own prompt (a recurrent
|
||
cell stores only the last position), so with `--ctx-checkpoints 0` there is no
|
||
checkpoint to fall back on and the server logs `forcing full prompt
|
||
re-processing … (likely due to SWA or hybrid/recurrent memory)`. Consequences:
|
||
intra-slot reuse only survives when the request echoes the assistant's
|
||
previous reply verbatim (a pure user resend is always a full re-prefill even
|
||
at f_sim 1.000); the RAM prompt cache becomes the ONLY carrier of a
|
||
conversation state ACROSS slots, so a cap that refuses deep states costs
|
||
minutes of re-prefill whenever slots are reassigned; state restoration only
|
||
runs via the LRU path (or LCP with f_keep < 0.5) — a plain prefix hit never
|
||
calls `prompt_load`.
|
||
|
||
**Verify a change:** live argv carries the flag (not the deploy log); `/health`
|
||
200 + real completion; VRAM unchanged and host RAM mostly free; no new
|
||
`exceeds cache size limit` lines, with a `making room` entry larger than the
|
||
old cap as direct proof. If the ceiling itself was not exercised live, say so —
|
||
crossing it needs a >120k-token conversation and must not run beside a live
|
||
service (bench law).
|
||
|
||
## 4. QSA flash-attn assert fix (qwen4exp)
|
||
|
||
`--flash-attn on` aborts graph build for qwen4exp models:
|
||
`GGML_ASSERT(a->type == GGML_TYPE_F32)` at `ggml.c:5371` through
|
||
`build_attn_qsa`. Root cause: `build_attn_qsa` calls `ggml_fill` on the
|
||
F16-cast kq mask returned by `get_kq_mask()` when flash attn is on, but
|
||
`ggml_fill` requires F32.
|
||
|
||
Fix (mirrors the MLA sparse path in `llm_graph_context::build_attn`): run the
|
||
top-k mask surgery on the F32 base mask (`inp->self_kq_mask`), cast to F16
|
||
only at the attention call when `flash_attn` is enabled
|
||
(`kq_mask_attn = cparams.flash_attn ? ggml_cast(ctx0, kq_mask_top_k,
|
||
GGML_TYPE_F16) : kq_mask_top_k`). The patch lives at
|
||
`scripts/patches/qwen4exp-qsa-f32-mask.patch`.
|
||
|
||
Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
|
||
it; CIRU, upstream and stock builds do not. Validate live: load with
|
||
`--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
|
||
clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
|
||
in the timings. The context cap that build reached for full-model residency is
|
||
part of that era's record on [[frank-dragoon]].
|
||
|
||
## Related
|
||
|
||
- [[GGUF Compatibility]] — what a payload is before it is priced
|
||
- [[frank-zealot]] — the endpoint whose Eras rows are the two-boot priced shapes this method produced
|
||
- [[frank-medium-llamacpp-dflash2]] — the state-law worked example behind section 3
|
||
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
|
||
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
|