docs: record dual-f16 KV refit on frank-default-small (2026-08-14)
This commit is contained in:
Frank
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`q8_0`, V=`f16`. Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~21.05 GB VRAM used, leaving an estimated **~3.0 GiB** headroom (chosen for prefill stability, see [[frank-default-small-benchmark]]). Idle RAM ~32 GB of 62 GB.
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB.
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM is close to saturated with this config. At idle ~21.05 GB used, leaving an estimated **~3.0 GiB** headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072`. A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
|
||||
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
@@ -102,6 +102,14 @@ It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
|
||||
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
|
||||
process RSS, process VSZ, and systemd MemoryPeak.
|
||||
|
||||
## Resolution: dual-f16 KV Cache (2026-08-14)
|
||||
|
||||
The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
|
||||
|
||||
The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
|
||||
|
||||
Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
|
||||
|
||||
## Mitigation Options
|
||||
|
||||
The documentation for the benchmark ([[frank-default-small-benchmark]])
|
||||
|
||||
+1
-1
@@ -143,7 +143,7 @@ The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B u
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
|
||||
|
||||
|
||||
@@ -76,17 +76,21 @@ Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
||||
|
||||
## VRAM Budget Model
|
||||
|
||||
The live service holds a fixed 800000-token KV pool plus the mmproj vision
|
||||
tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
|
||||
24 GiB budget:
|
||||
The live service holds a fixed 700416-token KV pool (f16/f16 since
|
||||
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
|
||||
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
|
||||
budget:
|
||||
|
||||
```
|
||||
service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
|
||||
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
|
||||
```
|
||||
|
||||
This model reproduces the historical OOMs. Configurations whose sum exceeds
|
||||
24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
|
||||
and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
|
||||
This model reproduces the historical OOMs. Configurations whose sum
|
||||
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
|
||||
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
|
||||
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
|
||||
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
|
||||
service runs 700416 cells (13,680 MiB).
|
||||
|
||||
## Candidate Configurations
|
||||
|
||||
@@ -111,7 +115,10 @@ moves one expert layer to CPU for stability:
|
||||
```
|
||||
|
||||
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
||||
44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
|
||||
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
|
||||
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
|
||||
removes the context-proportional flash-attn F16-conversion workspace that
|
||||
caused the cuBLAS crash class).
|
||||
|
||||
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
||||
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
||||
|
||||
Reference in New Issue
Block a user