---
type: reference
title: Dankserver VRAM and RAM Growth Under Load
description: Investigation of the recurring cuBLAS crash on frank-default-small. What grows, what does not, and what to do next.
tags:
  - dankserver
  - llamacpp
  - vram
  - ram
  - ctx-checkpoints
  - crash
---

# Dankserver VRAM and RAM Growth Under Load

This page records the investigation of the recurring cuBLAS crash on
`frank-default-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
with `CUDA error: an unsupported value or parameter was passed to the
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
at 19:45 and 19:55 and on 2026-08-14 at 12:00:43.

## What Was Ruled Out

### ctx-checkpoints do not grow VRAM

`--ctx-checkpoints 4` stores context checkpoints in host RAM. The log
records every checkpoint at exactly 62.813 MiB. The size never grows.
The reasoning:

- `common_prompt_checkpoint` (common/common.h) stores the serialized KV
  state in `std::vector<uint8_t>` fields. These live on the host heap.
- `update_tgt()` and `update_dft()` call `llama_state_seq_get_data_ext`
  with `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY`. The serialized state moves
  from VRAM to RAM for the snapshot and back on restore.
- With 4 slots and `--ctx-checkpoints 4`, the maximum checkpoint RAM is
  16 x 62.813 MiB, about 1.0 GiB. That is fixed and small.
- `data_spec` (spec-decode state) is empty for ngram-mod. The ngram-mod
  implementation does not override `get_state()`.

### ngram-mod does not grow VRAM

ngram-mod allocates one fixed 64 MiB CPU hash table
(`common_ngram_mod::entries`, 16M int32 entries). It never resizes. The
matcher keeps about 12 bytes per sequence. Draft generation runs on the
CPU. The speculative path allocates zero GPU memory.

### The CUDA graph cache has eviction in this build

`ggml_backend_cuda_context::cuda_graph()` sweeps every 5 seconds and
evicts graphs unused for 10 seconds. Upstream issue #25082 describes an
unbounded graph cache, but build b10108-13-g555881ebc includes the
eviction sweep. This build does not exhibit that leak.

## What Grows

### RAM growth is suspected, not yet confirmed

systemd reported MemoryPeak 46.9 GiB for the crashed process and 45.9
GiB for the replacement. RSS is flat at 32.3 GiB, which matches the
documented idle baseline of about 32 GiB. The difference is file-backed
page cache from the `--no-mmap` model read; cgroup memory.current counts
file pages attributed to the process. That gap is not proof of a leak by
itself.

The upstream pattern to watch is issue #26216: the prompt cache and
checkpoint accumulation exceed the `--cache-ram` cap and do not plateau.
The environment file does not set `--cache-ram`, so the llama.cpp
default of 8192 MiB applies. The monitor logs RSS, which separates
anonymous memory from file cache. If RSS climbs under load while VRAM is
flat, the RAM-side accumulation is real.

### VRAM suspects under load

VRAM is flat at idle: 21,268 MiB across every monitor sample. The crash
correlates with load density and slot context size (143K tokens on
slot 2), not with wall-clock time. The two mechanisms worth testing:

1. Flash-attention F16 K conversion workspace. The prefill kernel on
   Ampere is `MMA_F16`. It converts the quantized K cache (q8_0) to F16
   for the attention math. The workspace size scales with the attended
   context length (`ggml_nelements(K) x 2`). It is allocated into the
   graph buffer via `ggml_cuda_flash_attn_ext_get_alloc_size`.
2. The CUDA VMM pool grows monotonically. `ggml_cuda_pool_vmm::alloc`
   only reserves more physical memory; `free` only releases the
   reservation slot. The pool never shrinks during the process
   lifetime. Any new peak allocation permanently expands VRAM usage.

Both match the observed pattern: growth under prefill bursts at deep
context, flat at idle, crash when a burst exceeds the ~2.8 GiB headroom.

## Monitoring

The monitor runs on Dankserver:

```
/home/dank/vram-monitor.sh
/home/dank/vram-log.csv
```

It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
process RSS, process VSZ, and systemd MemoryPeak.

## Mitigation Options

The documentation for the benchmark ([[frank-default-small-benchmark]])
prescribes the first option for this crash class. None of these change
the model.

| Option | Effect | Cost |
| ------ | ------ | ---- |
| `GGML_CUDA_FORCE_MMQ=1` | Bypasses the cuBLAS path in the crashing function | Slight decode slowdown |
| `--cache-ram 2048` | Bounds the confirmed RAM growth | Cache misses on prompt reuse |
| `--ctx-checkpoints 2` | Cuts checkpoint RAM and save cost in half | More prefill work after eviction |
| Apply PR #24649 | Clears slot checkpoints before the prompt-cache save | Needs a llama.cpp rebuild |

Upstream references: issue #26216 (prompt cache growth), issue #25082
(graph cache, not applicable to this build), issue #23635 (quantized KV
with speculative decoding), PR #24649 (checkpoint-to-cache duplication),
PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
(same cuBLAS signature, confirmed on V100 with speculative decoding).

## See Also

- [[frank-default-small-benchmark]]
- [[Dankserver System Specs]]
- [[deployment]]