frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard

Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.

AGENTS.md

@@ -189,7 +189,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
 1. Stop the service before any benchmark:
    `sudo systemctl stop frank-default-small` (Dankserver) or
    `sudo systemctl stop frank-default-large` (Dankclaw).
-2. Run the bench on a scratch port (never the LLM service port 7770).
+2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
+   Scratch ports are not used for tests or benches on the LLM hosts — a
+   test process cannot share the single GPU with the running service anyway
+   (the model + KV fill the card), so changes are validated directly against
+   the service itself.
 3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
 4. Verify health before reporting:
    `curl http://192.168.0.235:7770/health` or
@@ -406,7 +410,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
 | Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
 | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
 | Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
-| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
+| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
 | Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |

docs/LLM/ikllama-ornith-dankserver.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: ik_llama.cpp on Dankserver (Ornith Q8_0)
-description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, q6_0 KV cache with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 160k single slot, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
+description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K q8_0 / V q5_0 with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 160k ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
 tags:
   - ikllama
   - llamacpp
@@ -23,7 +23,7 @@ timestamp: 2026-08-26
 
 The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
 3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
-encoder). Served as a **single 160000-context slot** (`PARALLEL=1`).
+encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
 Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
 model runbook.
 
@@ -105,14 +105,16 @@ Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
 `scripts/frank-default-small.service` in the repo. The service just calls
 the serve script; the env file holds all tunables.
 
-### KV cache: q8_0 → q6_0 + Hadamard
+### KV cache: K q8_0 / V q5_0 + Hadamard
 
-- `CACHE_TYPE_K=q6_0`, `CACHE_TYPE_V=q6_0` — 6-bit k-quant KV cache
+- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q5_0` — asymmetric KV precision: the
-  (passes `--cache-type-k q6_0 --cache-type-v q6_0`).
+  K cache at 8-bit, the V cache at 5-bit (passes `--cache-type-k q8_0
+  --cache-type-v q5_0`). Keeping K higher-precision than V is a common
+  asymmetric budget split.
 - `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
   `--v-cache-hadamard`. The Hadamard transform preserves KV quality at
-  6-bit. Implemented for both CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK)
+  quantized precision. Implemented for both CUDA (`ggml-cuda/hadamard.cu`)
-  paths. Flash-attention is **on by default**.
+  and CPU (IQK) paths. Flash-attention is **on by default**.
 
 > **KV cache type history on this build (`b166e269`):**
 > - `iq5_k` is **not** a KV-cache type — the cache whitelist is
@@ -122,10 +124,12 @@ the serve script; the env file holds all tunables.
 > - `q8_KV` **crashes** with a `ggml view` size assert
 >   (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
 >   QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
-> - **`q6_0` + Hadamard works** and is what this deployment uses.
+> - `q6_0` (both caches) boots and served, but with `PARALLEL=3` the KV
+>   tripled, so the K cache went back up to `q8_0` and V down to `q5_0` for
+>   the chosen K/V budget split.
 
-At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving
+With 3 parallel slots at 160k context, the KV cache is ~3x a single slot's.
-~2.2 GiB headroom on the 24 GiB card after the GPU-side experts.
+The GPU-side expert set is reduced to fit (see the expert section).
 
 ### Split mode: none (single GPU)
 
@@ -148,15 +152,17 @@ At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving
 
 The expert-offload pattern from upstream llama.cpp carries over: the
 `--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
-the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
+the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
-On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA
+Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
-(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the
+MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
-GPU headroom for lower latency; the rest of that CPU range stays on CPU.
+are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
+quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
+no exps and are untouched by the regex.
 
-- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
+- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
   experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
-  24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention +
+  24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
-  35-40 set) are left off the list so their experts load to CUDA.
+  full-attention set) are left off the list so their experts load to CUDA.
 - `OVERRIDE_BUFT=CPU`.
 - `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
   integer (the literal `all` is rejected). 999 means "load as many tensor
@@ -169,10 +175,9 @@ GPU headroom for lower latency; the rest of that CPU range stays on CPU.
 > boot log for the running service.
 
 To move more experts back to the GPU later, drop more block numbers from
-`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
+`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
-first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the
+headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
-free headroom was ~1.6 GiB, which is tighter than ideal — add more only
+cache there is little headroom, so the default keeps only 35-40 on GPU.
-with a headroom check.
 
 ### Qwen-Sharp chat template via --jinja
 

scripts/frank-default-small.env

@@ -5,8 +5,8 @@
 # 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
 # RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
 # pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
-# KV cache quantized to iq5_k with Hadamard transforms on both K and V.
+# KV cache K q8_0 / V q5_0 with Hadamard transforms on both (see the KV
-# Split mode none (single GPU; graph would require 2+ devices).
+# section below). Split mode none (single GPU; graph would require 2+).
 #
 # Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
 # ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
@@ -43,40 +43,41 @@ ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
 
-# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative
+# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama
-# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE
+# limits speculative decoding to a single slot, so SPEC stays none (MTP is
-# =none) unless a user asks to re-enable ngram-mod/MTP.
+# also disabled). The 3 slots triple the KV cache, so enough expert weights
+# are CPU-pinned for the cache to fit on the 24 GB card.
 CTX_SIZE=160000
-PARALLEL=1
+PARALLEL=3
 
-# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays
+# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at
-# off for the low-parameter MoE (3B active of 35B). MTP is also disabled.
+# PARALLEL=3 a configured spec stage hard-exits at load. The low-parameter
+# MoE (3B active of 35B) does not need it anyway. MTP is also disabled.
 SPEC_TYPE=none
 
-# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
+# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q8_0
-# layer audit found 630 MiB of expert tensors per block. Keep every
+# audit found ~1088 MiB of expert tensors per block (4 tensor × 272 MiB). Keep
-# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
+# every final/output-adjacent block 35-40 on CUDA; full-attention blocks 3,7,
-# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
+# 11,15,19,23 have no exps and are untouched by this regex. All other expert
-# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
+# tensors are pinned to CPU. Buffer type "CPU" is the CPU buft on this host.
-# CPU buft name on this host.
 #
-# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
+# 2026-08-26: PARALLEL=3 (CTX 160000) triples the KV cache, so the blocks
-# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB)
+# that were moved back to CUDA for fused-MoE (27-34) are pinned to CPU again
-# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU
+# to free GPU. Blocks 35-40 stay on CUDA. If it boot-loops, move the next
-# range below those stays on CPU.
+# lower block into the CPU regex one at a time.
-OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 # ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
 # mean "load as many layers as possible". Expert exps are pinned to CPU by
 # OVERRIDE_EXPERT below.
 N_GPU_LAYERS=999
 
-# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both
+# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with
-# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build
+# Hadamard transforms. The V cache at lower precision than K is a common
-# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we
+# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c
-# verified works. Flash-attention is on by default.
+# :5427). Flash-attention is on by default.
-CACHE_TYPE_K=q6_0
+CACHE_TYPE_K=q8_0
-CACHE_TYPE_V=q6_0
+CACHE_TYPE_V=q5_0
 K_CACHE_HADAMARD=1
 V_CACHE_HADAMARD=1
 

scripts/serve-frank-small.sh

@@ -5,11 +5,12 @@
 #
 # Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
 # 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
-# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
+# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
+# is disabled (ik_llama limits spec to a single slot).
 #
-# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
+# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
-# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
+# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
-# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
+# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
 # Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
 # split-mode graph with fewer than 2 devices (hard load error).
 
@@ -49,10 +50,10 @@ fi
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=160000}"
-: "${PARALLEL:=1}"
+: "${PARALLEL:=3}"
 : "${SPEC_TYPE:=none}"
-: "${CACHE_TYPE_K:=q6_0}"
+: "${CACHE_TYPE_K:=q8_0}"
-: "${CACHE_TYPE_V:=q6_0}"
+: "${CACHE_TYPE_V:=q5_0}"
 : "${K_CACHE_HADAMARD:=0}"
 : "${V_CACHE_HADAMARD:=0}"
 : "${SPLIT_MODE:=none}"

tests/frank_small_config.bats

@@ -14,41 +14,46 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
     grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE"
 }
 
-@test "frank-default-small uses a single 160000 context slot" {
+@test "frank-default-small uses three 160000 context slots" {
     grep -q '^CTX_SIZE=160000$' "$ENV_FILE"
-    grep -q '^PARALLEL=1$' "$ENV_FILE"
+    grep -q '^PARALLEL=3$' "$ENV_FILE"
 }
 
-@test "frank-default-small uses q6_0 KV cache with Hadamard" {
+@test "frank-default-small uses K q8_0 + V q5_0 KV cache with Hadamard" {
-    grep -q '^CACHE_TYPE_K=q6_0$' "$ENV_FILE"
+    grep -q '^CACHE_TYPE_K=q8_0$' "$ENV_FILE"
-    grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE"
+    grep -q '^CACHE_TYPE_V=q5_0$' "$ENV_FILE"
     grep -q '^K_CACHE_HADAMARD=1$' "$ENV_FILE"
     grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE"
 }
 
 @test "frank-default-small keeps full-attention and final blocks on CUDA" {
     source "$ENV_FILE"
-    [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
+    [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
+    # output-adjacent final blocks stay on CUDA
+    [[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|37|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|38|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|39|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
+    # full-attention blocks (3,7,11,15,19,23) are untouched (no exps)
     [[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|27|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|31|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
     [[ "$OVERRIDE_BUFT" == CPU ]]
     # ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
     # loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
     [[ "$N_GPU_LAYERS" == 999 ]]
 }
 
-@test "frank-default-small disables speculative decoding (single slot)" {
+@test "frank-default-small disables speculative decoding (3 parallel slots)" {
+    # ik_llama limits speculative decoding to -np 1; with PARALLEL=3 a
+    # configured spec stage hard-exits, so SPEC_TYPE=none.
     grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
-    grep -q '^PARALLEL=1$' "$ENV_FILE"
+    grep -q '^PARALLEL=3$' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"