From 6663af1041770676dfcf8ae142fae03cf4e2c81c Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Thu, 13 Aug 2026 17:03:45 -0700
Subject: [PATCH] Revert "Remove KV cache quantization, increase batch/ubatch
 on frank-default-large"

The Aug 13 change crashed frank-default-large in a restart loop. The service
segfaulted ~15s into model load (SIGSEGV in hipMalloc during load). The
benchmark (frank-default-large-benchmark.md) already proved ubatch 2048 OOMs
(40.9GB prefill buffer) and batch 2048 / ubatch 512 is the fits-and-loads
config on the 96GiB UMA budget. Restore q8_0/q8_0 KV cache (FA pairing) and
batch 2048 / ubatch 512.

Also fix deployment.md: KV is q8_0 K / q8_0 V (not f16 V) since 334b6af.
---
 docs/deployment.md                 |  2 +-
 scripts/frank-default-large.env    | 18 +++++++++++++-----
 scripts/serve-llama-frank-large.sh |  8 ++++++--
 3 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/docs/deployment.md b/docs/deployment.md
index c9ba79f..c5bc0bb 100644
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -100,7 +100,7 @@ curl -sf http://127.0.0.1:7770/health
 
 ### Dankclaw — frank-default-large
 
-The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `f16` V on GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
+The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
 
 > Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices.
 
diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env
index c4b869e..b8b6ca7 100644
--- a/scripts/frank-default-large.env
+++ b/scripts/frank-default-large.env
@@ -20,11 +20,19 @@ SPEC_NGRAM_MOD_N_MATCH=24
 DEVICE=ROCm0
 # Dense model (Qwen3.6-27B, not MoE): no MoE CPU offload. Full offload.
 N_GPU_LAYERS=all
-# KV on GPU, fp16 (unquantized). Default ggml_type for both K and V.
-# Batch sizing: fp16 KV is larger than q8_0, but user has verified this
-# configuration fits. See frank-default-large-benchmark.md.
-BATCH_SIZE=8192
-UBATCH_SIZE=2048
+# KV on GPU, quantized (K q8_0, V q8_0). V MUST match K: this llama.cpp
+# build has GGML_CUDA_FA_ALL_QUANTS=OFF, and Flash Attention requires
+# K type == V type. Mixed q8_0/f16 silently disables FA, collapsing
+# long-context decode to ~1 t/s at 200k ctx. q8_0 V halves V memory too.
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q8_0
+# Batch sizing constrained by ROCm memory at 602100 ctx (3x200700):
+# KV ~50GB + model 29.77GB leave only ~16GB for compute graphs. The compute
+# graph scales with UBATCH, so ubatch must stay ~512; ubatch 2048 OOMs even
+# at batch 2048. Proven-fits config: batch 2048 / ubatch 512.
+# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
+BATCH_SIZE=2048
+UBATCH_SIZE=512
 # Sampling settings — per HauhauCS Qwen3.6-27B model card.
 # Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
 # Clients may override per-request; these are the server-side defaults.
diff --git a/scripts/serve-llama-frank-large.sh b/scripts/serve-llama-frank-large.sh
index c088350..30e4b20 100755
--- a/scripts/serve-llama-frank-large.sh
+++ b/scripts/serve-llama-frank-large.sh
@@ -33,8 +33,10 @@ fi
 : "${SPEC_NGRAM_MOD_N_MATCH:=24}"
 : "${DEVICE:=ROCm0}"
 : "${N_GPU_LAYERS:=all}"
-: "${BATCH_SIZE:=8192}"
-: "${UBATCH_SIZE:=2048}"
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q8_0}"
+: "${BATCH_SIZE:=2048}"
+: "${UBATCH_SIZE:=512}"
 : "${TEMP:=1.0}"
 : "${TOP_K:=20}"
 : "${TOP_P:=0.95}"
@@ -64,6 +66,8 @@ exec "$LLAMA_SERVER" \
     --device "$DEVICE" \
     --gpu-layers "$N_GPU_LAYERS" \
     --fit off \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
     --ctx-size "$CTX_SIZE" \
     --parallel "$PARALLEL" \
     --batch-size "$BATCH_SIZE" \