frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard

Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.

scripts/frank-default-small.env

@@ -5,8 +5,8 @@
 # 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
 # RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
 # pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
-# KV cache quantized to iq5_k with Hadamard transforms on both K and V.
-# Split mode none (single GPU; graph would require 2+ devices).
+# KV cache K q8_0 / V q5_0 with Hadamard transforms on both (see the KV
+# section below). Split mode none (single GPU; graph would require 2+).
 #
 # Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
 # ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
@@ -43,40 +43,41 @@ ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
 
-# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative
-# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE
-# =none) unless a user asks to re-enable ngram-mod/MTP.
+# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama
+# limits speculative decoding to a single slot, so SPEC stays none (MTP is
+# also disabled). The 3 slots triple the KV cache, so enough expert weights
+# are CPU-pinned for the cache to fit on the 24 GB card.
 CTX_SIZE=160000
-PARALLEL=1
+PARALLEL=3
 
-# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays
-# off for the low-parameter MoE (3B active of 35B). MTP is also disabled.
+# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at
+# PARALLEL=3 a configured spec stage hard-exits at load. The low-parameter
+# MoE (3B active of 35B) does not need it anyway. MTP is also disabled.
 SPEC_TYPE=none
 
-# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
-# layer audit found 630 MiB of expert tensors per block. Keep every
-# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
-# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
-# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
-# CPU buft name on this host.
+# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q8_0
+# audit found ~1088 MiB of expert tensors per block (4 tensor × 272 MiB). Keep
+# every final/output-adjacent block 35-40 on CUDA; full-attention blocks 3,7,
+# 11,15,19,23 have no exps and are untouched by this regex. All other expert
+# tensors are pinned to CPU. Buffer type "CPU" is the CPU buft on this host.
 #
-# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
-# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB)
-# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU
-# range below those stays on CPU.
-OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+# 2026-08-26: PARALLEL=3 (CTX 160000) triples the KV cache, so the blocks
+# that were moved back to CUDA for fused-MoE (27-34) are pinned to CPU again
+# to free GPU. Blocks 35-40 stay on CUDA. If it boot-loops, move the next
+# lower block into the CPU regex one at a time.
+OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 # ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
 # mean "load as many layers as possible". Expert exps are pinned to CPU by
 # OVERRIDE_EXPERT below.
 N_GPU_LAYERS=999
 
-# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both
-# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build
-# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we
-# verified works. Flash-attention is on by default.
-CACHE_TYPE_K=q6_0
-CACHE_TYPE_V=q6_0
+# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with
+# Hadamard transforms. The V cache at lower precision than K is a common
+# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c
+# :5427). Flash-attention is on by default.
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q5_0
 K_CACHE_HADAMARD=1
 V_CACHE_HADAMARD=1
 

scripts/serve-frank-small.sh

@@ -5,11 +5,12 @@
 #
 # Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
 # 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
-# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
+# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
+# is disabled (ik_llama limits spec to a single slot).
 #
-# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
-# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
-# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
+# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
+# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
+# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
 # Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
 # split-mode graph with fewer than 2 devices (hard load error).
 
@@ -49,10 +50,10 @@ fi
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=160000}"
-: "${PARALLEL:=1}"
+: "${PARALLEL:=3}"
 : "${SPEC_TYPE:=none}"
-: "${CACHE_TYPE_K:=q6_0}"
-: "${CACHE_TYPE_V:=q6_0}"
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q5_0}"
 : "${K_CACHE_HADAMARD:=0}"
 : "${V_CACHE_HADAMARD:=0}"
 : "${SPLIT_MODE:=none}"