docs: record dual-f16 KV refit on frank-default-small (2026-08-14)

docs/Dankserver System Specs.md

@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
 
 ### Active Services
 
-The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`q8_0`, V=`f16`. Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~21.05 GB VRAM used, leaving an estimated **~3.0 GiB** headroom (chosen for prefill stability, see [[frank-default-small-benchmark]]). Idle RAM ~32 GB of 62 GB.
+The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB.
 
 ## Hardware Relevance
 
-- 24 GB VRAM is close to saturated with this config. At idle ~21.05 GB used, leaving an estimated **~3.0 GiB** headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072`. A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
+- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
 - 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
 - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
 - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).

docs/dankserver-vram-growth.md

@@ -102,6 +102,14 @@ It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
 row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
 process RSS, process VSZ, and systemd MemoryPeak.
 
+## Resolution: dual-f16 KV Cache (2026-08-14)
+
+The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
+
+The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
+
+Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
+
 ## Mitigation Options
 
 The documentation for the benchmark ([[frank-default-small-benchmark]])

docs/deployment.md

@@ -143,7 +143,7 @@ The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B u
 
 ### Dankserver — frank-default-small
 
-The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
+The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
 
 > Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
 

docs/frank-default-small-benchmark.md

@@ -76,17 +76,21 @@ Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
 
 ## VRAM Budget Model
 
-The live service holds a fixed 800000-token KV pool plus the mmproj vision
+The live service holds a fixed 700416-token KV pool (f16/f16 since
-tower plus CUDA overhead. The model below fits llama-bench peak VRAM into a
+2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
-24 GiB budget:
+overhead. The model below fits llama-bench peak VRAM into a 24 GiB
+budget:
 
 ```
-service VRAM = llama-bench peak + KV pool (11.44 GiB) + overhead (2.2 GiB)
+service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
 ```
 
-This model reproduces the historical OOMs. Configurations whose sum exceeds
+This model reproduces the historical OOMs. Configurations whose sum
-24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about 24.4 GiB
+exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
-and fails, which matches the recorded OOMs at `ncmoe 32` and `33`.
+24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
+`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
+800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
+service runs 700416 cells (13,680 MiB).
 
 ## Candidate Configurations
 
@@ -111,7 +115,10 @@ moves one expert layer to CPU for stability:
 ```
 
 This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
-44.4 t/s, and about 3.0 GiB of headroom at the 800000-token pool.
+44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
+(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
+removes the context-proportional flash-attn F16-conversion workspace that
+caused the cuBLAS crash class).
 
 `ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
 `ubatch 3072`, a big prefill burst failed a cuBLAS call under memory