frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache

Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.

Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
  --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
  (legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
  pinned to CPU via --override-tensor

Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.

scripts/frank-default-small.env

@@ -1,11 +1,18 @@
 # Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q6_K, MTP MoE,
 # Dankserver, RTX 3090).
 #
-# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen
-# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit
-# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS
-# Qwen3.6-35B-A3B model on 2026-08-24. Switched from CRACK Q8_0 to Q6_K on
-# 2026-08-25.
+# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp, built
+# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
+# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
+# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
+# KV cache quantized to q8_KV with Hadamard transforms on both K and V.
+# --split-mode graph enabled (single GPU; retained per operator preference).
+#
+# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
+# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
+# so the q8_KV + Hadamard KV cache combos compile into the CUDA/CPU FA
+# kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24.
+# Switched from CRACK Q8_0 to Q6_K on 2026-08-25.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
 # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
@@ -16,8 +23,12 @@
 MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
 MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf
 MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
-# Upstream llama.cpp binary (b10121).
-LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
+# ik_llama.cpp llama-server (built 2026-08-26 from ikawrakow/ik_llama.cpp @
+# b166e269; CUDA 13.2, -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_FA_ALL_QUANTS=ON,
+# arch 86 for the RTX 3090). Uses the staged pip cu13 toolkit at
+# /home/dank/cuda13-stage. Rollback: switch to
+# /home/dank/llama.cpp/build/bin/llama-server (upstream b10121).
+LLAMA_SERVER=/home/dank/ik_llama.cpp/build/bin/llama-server
 
 # Qwen-Sharp chat template from peculiar-ragdoll (tracked in the repo,
 # staged on the host).
@@ -35,10 +46,13 @@ CTX_SIZE=491520
 PARALLEL=3
 
 # Speculative decoding: ngram-mod only. MTP is intentionally disabled.
-SPEC_TYPE=ngram-mod
-SPEC_NGRAM_MOD_N_MIN=48
-SPEC_NGRAM_MOD_N_MAX=64
-SPEC_NGRAM_MOD_N_MATCH=24
+# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
+# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
+#   old --spec-ngram-mod-n-min  (48) -> n_min
+#   old --spec-ngram-mod-n-max  (64) -> n_max
+#   old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
+# SPEC_TYPE is passed through to --spec-type.
+SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
 
 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
 # layer audit found 630 MiB of expert tensors per block. Keep every
@@ -48,11 +62,23 @@ SPEC_NGRAM_MOD_N_MATCH=24
 # CPU buft name on this host.
 OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
-N_GPU_LAYERS=all
+# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
+# mean "load as many layers as possible". Expert exps are pinned to CPU by
+# OVERRIDE_EXPERT below.
+N_GPU_LAYERS=999
 
-# KV cache on GPU, both K and V at f16.
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
+# KV cache on GPU, both K and V at q8_KV (fast quant type on ik_llama, PR 208)
+# plus Hadamard transforms on both caches to preserve quality at 8-bit.
+# For quantized KV below Q6_0 the docs recommend --k-cache-hadamard; we apply
+# --v-cache-hadamard to V as well.
+CACHE_TYPE_K=q8_KV
+CACHE_TYPE_V=q8_KV
+K_CACHE_HADAMARD=1
+V_CACHE_HADAMARD=1
+
+# Split mode: graph (multi-GPU graph split; on a single 3090 this is a no-op
+# for correctness but kept per operator preference).
+SPLIT_MODE=graph
 
 VERBOSITY=4
 # Store context checkpoints in RAM to hold peak GPU memory down.

scripts/serve-frank-small.sh

@@ -1,11 +1,19 @@
 #!/usr/bin/env bash
 # Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as
-# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE
+# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE
 # experts on CPU).
 #
-# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
-# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this
-# service uses ngram-mod instead of MTP speculation.
+# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
+# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
+# this service uses ngram-mod instead of MTP speculation.
+#
+# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
+# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
+#
+# Split mode graph + --override-tensor CPU expert offload. Per ik_llama
+# README, `--split-mode graph` + partial GPU offload can occasionally produce
+# incoherent output; the documented fallback is `-cuda graphs=0`. Smoke-test
+# after each model load.
 #
 # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
 # --override-tensor pattern are pinned to CPU. Selected full-attention and
@@ -38,19 +46,19 @@ fi
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=491520}"
 : "${PARALLEL:=3}"
-: "${SPEC_TYPE:=ngram-mod}"
-: "${SPEC_NGRAM_MOD_N_MIN:=48}"
-: "${SPEC_NGRAM_MOD_N_MAX:=64}"
-: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
+: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
+: "${K_CACHE_HADAMARD:=0}"
+: "${V_CACHE_HADAMARD:=0}"
+: "${SPLIT_MODE:=none}"
 : "${VERBOSITY:=4}"
 # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
 # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
 # MoE expert weights and leave every other tensor on the GPU.
 : "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
 : "${OVERRIDE_BUFT:=CPU}"
-: "${N_GPU_LAYERS:=all}"
+: "${N_GPU_LAYERS:=999}"
 : "${BATCH_SIZE:=4096}"
 : "${UBATCH_SIZE:=1024}"
 : "${CTX_CHECKPOINTS:=16}"
@@ -81,12 +89,12 @@ exec "$LLAMA_SERVER" \
     --mmproj "$MMPROJ" \
     --chat-template-file "$CHAT_TEMPLATE_FILE" \
     --spec-type "$SPEC_TYPE" \
-    --spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \
-    --spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \
-    --spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \
     --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
     --cache-type-k "$CACHE_TYPE_K" \
     --cache-type-v "$CACHE_TYPE_V" \
+    $([ "$K_CACHE_HADAMARD" = "1" ] && printf -- "--k-cache-hadamard") \
+    $([ "$V_CACHE_HADAMARD" = "1" ] && printf -- "--v-cache-hadamard") \
+    --split-mode "$SPLIT_MODE" \
     --verbosity "$VERBOSITY" \
     --ctx-size "$CTX_SIZE" \
     --parallel "$PARALLEL" \