docs: record Dankserver VRAM/RAM growth investigation (2026-08-14)

docs/dankserver-vram-growth.md

@@ -0,0 +1,128 @@
+---
+type: reference
+title: Dankserver VRAM and RAM Growth Under Load
+description: Investigation of the recurring cuBLAS crash on frank-default-small. What grows, what does not, and what to do next.
+tags:
+  - dankserver
+  - llamacpp
+  - vram
+  - ram
+  - ctx-checkpoints
+  - crash
+---
+
+# Dankserver VRAM and RAM Growth Under Load
+
+This page records the investigation of the recurring cuBLAS crash on
+`frank-default-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
+with `CUDA error: an unsupported value or parameter was passed to the
+function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
+`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
+at 19:45 and 19:55 and on 2026-08-14 at 12:00:43.
+
+## What Was Ruled Out
+
+### ctx-checkpoints do not grow VRAM
+
+`--ctx-checkpoints 4` stores context checkpoints in host RAM. The log
+records every checkpoint at exactly 62.813 MiB. The size never grows.
+The reasoning:
+
+- `common_prompt_checkpoint` (common/common.h) stores the serialized KV
+  state in `std::vector<uint8_t>` fields. These live on the host heap.
+- `update_tgt()` and `update_dft()` call `llama_state_seq_get_data_ext`
+  with `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY`. The serialized state moves
+  from VRAM to RAM for the snapshot and back on restore.
+- With 4 slots and `--ctx-checkpoints 4`, the maximum checkpoint RAM is
+  16 x 62.813 MiB, about 1.0 GiB. That is fixed and small.
+- `data_spec` (spec-decode state) is empty for ngram-mod. The ngram-mod
+  implementation does not override `get_state()`.
+
+### ngram-mod does not grow VRAM
+
+ngram-mod allocates one fixed 64 MiB CPU hash table
+(`common_ngram_mod::entries`, 16M int32 entries). It never resizes. The
+matcher keeps about 12 bytes per sequence. Draft generation runs on the
+CPU. The speculative path allocates zero GPU memory.
+
+### The CUDA graph cache has eviction in this build
+
+`ggml_backend_cuda_context::cuda_graph()` sweeps every 5 seconds and
+evicts graphs unused for 10 seconds. Upstream issue #25082 describes an
+unbounded graph cache, but build b10108-13-g555881ebc includes the
+eviction sweep. This build does not exhibit that leak.
+
+## What Grows
+
+### RAM growth is suspected, not yet confirmed
+
+systemd reported MemoryPeak 46.9 GiB for the crashed process and 45.9
+GiB for the replacement. RSS is flat at 32.3 GiB, which matches the
+documented idle baseline of about 32 GiB. The difference is file-backed
+page cache from the `--no-mmap` model read; cgroup memory.current counts
+file pages attributed to the process. That gap is not proof of a leak by
+itself.
+
+The upstream pattern to watch is issue #26216: the prompt cache and
+checkpoint accumulation exceed the `--cache-ram` cap and do not plateau.
+The environment file does not set `--cache-ram`, so the llama.cpp
+default of 8192 MiB applies. The monitor logs RSS, which separates
+anonymous memory from file cache. If RSS climbs under load while VRAM is
+flat, the RAM-side accumulation is real.
+
+### VRAM suspects under load
+
+VRAM is flat at idle: 21,268 MiB across every monitor sample. The crash
+correlates with load density and slot context size (143K tokens on
+slot 2), not with wall-clock time. The two mechanisms worth testing:
+
+1. Flash-attention F16 K conversion workspace. The prefill kernel on
+   Ampere is `MMA_F16`. It converts the quantized K cache (q8_0) to F16
+   for the attention math. The workspace size scales with the attended
+   context length (`ggml_nelements(K) x 2`). It is allocated into the
+   graph buffer via `ggml_cuda_flash_attn_ext_get_alloc_size`.
+2. The CUDA VMM pool grows monotonically. `ggml_cuda_pool_vmm::alloc`
+   only reserves more physical memory; `free` only releases the
+   reservation slot. The pool never shrinks during the process
+   lifetime. Any new peak allocation permanently expands VRAM usage.
+
+Both match the observed pattern: growth under prefill bursts at deep
+context, flat at idle, crash when a burst exceeds the ~2.8 GiB headroom.
+
+## Monitoring
+
+The monitor runs on Dankserver:
+
+```
+/home/dank/vram-monitor.sh
+/home/dank/vram-log.csv
+```
+
+It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
+row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
+process RSS, process VSZ, and systemd MemoryPeak.
+
+## Mitigation Options
+
+The documentation for the benchmark ([[frank-default-small-benchmark]])
+prescribes the first option for this crash class. None of these change
+the model.
+
+| Option | Effect | Cost |
+| ------ | ------ | ---- |
+| `GGML_CUDA_FORCE_MMQ=1` | Bypasses the cuBLAS path in the crashing function | Slight decode slowdown |
+| `--cache-ram 2048` | Bounds the confirmed RAM growth | Cache misses on prompt reuse |
+| `--ctx-checkpoints 2` | Cuts checkpoint RAM and save cost in half | More prefill work after eviction |
+| Apply PR #24649 | Clears slot checkpoints before the prompt-cache save | Needs a llama.cpp rebuild |
+
+Upstream references: issue #26216 (prompt cache growth), issue #25082
+(graph cache, not applicable to this build), issue #23635 (quantized KV
+with speculative decoding), PR #24649 (checkpoint-to-cache duplication),
+PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
+(same cuBLAS signature, confirmed on V100 with speculative decoding).
+
+## See Also
+
+- [[frank-default-small-benchmark]]
+- [[Dankserver System Specs]]
+- [[deployment]]