frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache

Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.

Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
  --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
  (legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
  pinned to CPU via --override-tensor

Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.

docs/LLM/ikllama-ornith-dankserver.md

@@ -0,0 +1,208 @@
+---
+type: reference
+title: ik_llama.cpp on Dankserver (Ornith Q6_K)
+description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode graph, ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
+tags:
+  - ikllama
+  - llamacpp
+  - cuda
+  - dankserver
+  - moe
+  - ornith
+  - kv-cache
+  - hadamard
+timestamp: 2026-08-26
+---
+
+# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
+
+> **Active 2026-08-26.** `frank-default-small` switched from upstream
+> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
+> RTX 3090. This page documents the build, the mandatory FA build flags, and
+> the MoE / CPU-offload runtime arguments.
+
+The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
+Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
+Speculation stays ngram-only (MTP intentionally disabled). See
+[[dealign-ornith-crack]] for the model runbook.
+
+## Why ik_llama.cpp
+
+ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
+optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
+cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
+exactly the supported hybrid the fork targets. Only CPU and CUDA backends
+are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
+
+## Build recipe (CUDA 13.2 via staged pip toolkit)
+
+The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
+is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
+13.2 toolkit is installed via **pip** inside the vllm venv at
+`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
+self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
+`.so` aliases and static libs CMake needs.
+
+```bash
+# 1. Clone a pinned ik_llama.cpp
+git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
+cd /home/dank/ik_llama.cpp
+git checkout b166e269   # the version this deployment was built from
+
+# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
+cmake -B build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DGGML_CUDA=ON \
+  -DGGML_CUDA_FA_ALL_QUANTS=ON \
+  -DGGML_IQK_FA_ALL_QUANTS=ON \
+  -DCMAKE_CUDA_ARCHITECTURES=86 \
+  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
+  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
+  -DGGML_NATIVE=ON
+
+# 3. Build
+cmake --build build --config Release -j 8
+```
+
+### ⚠️ MANDATORY: always include the FA all-quants flags
+
+> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
+> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
+> BOTH:
+> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
+>   into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
+>   a subset and **errors at runtime** telling you to recompile when a combo
+>   (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
+>   the GPU, so this is the flag that matters here.
+> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
+>   IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
+>   FA kernels. Relevant because the MoE experts run on the CPU; the KV
+>   cache that backs CPU-side attention uses the IQK FA path.
+>
+> These are the old "asymmetrical quants" compile-time gates. On a CUDA
+> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
+> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
+
+`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
+`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
+verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
+
+### CUDA stage note
+
+`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
+(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
+and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
+CCCL CUDA version skew by defining
+`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
+package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
+a strict header lint). Do not re-run configure without that flag or you get
+a hard compile error.
+
+## Runtime configuration
+
+Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
+`scripts/frank-default-small.service` in the repo. The service just calls
+the serve script; the env file holds all tunables.
+
+### KV cache: q8_KV + Hadamard
+
+- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik
+  PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`.
+- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
+  `--v-cache-hadamard`. The Hadamard transform preserves KV quality at
+  8-bit (docs recommend it for quantized KV below Q6_0). Implemented for
+  both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths.
+
+These keep the 163840-slot / 491520-pool context while roughly halving KV
+cache RAM vs f16.
+
+### Split-mode graph
+
+- `SPLIT_MODE=graph` → `--split-mode graph`. On a single 3090 this is a
+  no-op for correctness but keeps the graph scheduled across the compute
+  buffers. ik recommends it for MoE when more than one device is present;
+  here it is retained per operator preference alongside the explicit
+  `--override-tensor` expert pins.
+
+> **Caveat:** the ik_llama README warns that `--split-mode graph` combined
+> with partial GPU offload (`--cpu-moe` / `--n-cpu-moe` / `--override-tensor`)
+> can rarely produce incoherent output. The documented mitigation is
+> `-cuda graphs=0`. Smoke-test after every load; if a response is gibberish,
+> add `-cuda graphs=0` to the serve line and re-test.
+
+### Expert offload (MoE on CPU)
+
+The expert-offload pattern is unchanged from upstream llama.cpp: the
+`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
+the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
+
+- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
+- `OVERRIDE_BUFT=CPU`.
+- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
+  integer (the literal `all` is rejected). 999 means "load as many tensor
+  layers as possible", while the expert override still pins the exps to CPU.
+
+### Speculative decoding: canonical ngram-mod
+
+ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
+flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
+- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`
+
+Mapping from the old upstream config:
+| legacy flag | old value | new canonical key |
+|-------------|----------|-------------------|
+| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
+| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
+| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |
+
+MTP remains intentionally disabled.
+
+### Flash-attention + sampling
+
+Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
+`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
+checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
+`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
+`NO_MMAP`) are unchanged and pass straight through.
+
+## MoE / hybrid-arg reference (not in upstream llama.cpp)
+
+Several ik_llama knobs matter for a CPU-MoE offload host:
+
+| flag | meaning | our use |
+|------|---------|---------|
+| `--split-mode graph` | split tensors+graph across devices; strong for MoE | retained (single-GPU no-op) |
+| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
+| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
+| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
+| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
+| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
+
+The current deployment uses `--override-tensor` (not `--fit` or
+`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
+apply.
+
+## Rollback
+
+The previous runtime `upstream llama.cpp b10121` and its binary
+(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
+rollback to the old stack, change `LLAMA_SERVER=` in
+`scripts/frank-default-small.env` back to that path and
+`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
+`--spec-ngram-mod-*` flags. The model file is not changed either way.
+
+## See Also
+
+- [[dealign-ornith-crack]] — the Ornith Q6_K model run
+- [[Dankserver System Specs]] — the host
+- [[deployment]] — service-scoped git deploy
+
+## Citations
+
+- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
+  MoE offload guidance and the `--split-mode graph` + CPU-offload warning
+- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
+  build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
+- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
+  q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
+  `--split-mode graph` MoE note (PR 1080)

docs/LLM/index.md

@@ -1,7 +1,7 @@
 # LLM Fleet — Knowledge Base
 
 Documentation for the LLM inference fleet on two local machines
-(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
+(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
 Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
 (frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
 services run the Qwen-Sharp chat template.
@@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template.
 graph LR
     subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
         A[RTX 3090 24GB + 62GB RAM]
-        B[upstream llama.cpp b10121
+        B2[ik_llama.cpp build b166e269
 frank-default-small, port 7770]
         C[Ornith-1.5-35B-A3B-CRACK Q6_K
-ngram-mod, selected experts offloaded via override, 491520 ctx]
+ngram-mod, q8_KV + Hadamard cache, 491520 ctx]
-        A --> B --> C
+        A --> B2 --> C
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
-        D[gfx1151 iGPU 64 GiB unified memory]
+        E2[gfx1151 iGPU 64 GiB unified memory]
         E[charlie12345/ROCmFPX build-kairic (v81)
 frank-default-large, port 7770]
         F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
 MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
-        D --> E --> F
+        E2 --> E --> F
     end
 
-    I[LLM Clients / API] -->|HTTP :7770| B
+    CLI[LLM Clients / API] -->|HTTP :7770| B2
-    I -->|HTTP :7770| E
+    CLI -->|HTTP :7770| E
 ```
 
 ## Servers
@@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
   on Dankclaw (retired)
 - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
   Dankserver (frank-default-small, active 2026-08-24)
+- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
+  frank-default-small (2026-08-26): q8_KV + Hadamard KV cache,
+  split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS
+  build flags
 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
   model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
-- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
-  Dankserver (frank-default-small, active 2026-08-24)
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
   on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment
   (deployed and reverted 2026-08-22)
-- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
+- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
 - [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
 
 ## CIRU Reference (Dankclaw)

docs/index.md

@@ -110,9 +110,10 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
+- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_KV + Hadamard KV cache, split-mode graph; 2026-08-26)
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
 - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
-- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
+- [[Dankserver System Specs]] — the RTX 3090 box hardware
 - [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
 - [[Dankclaw Improvements]] — host-level changes on the CIRU box
 

scripts/frank-default-small.env

@@ -1,11 +1,18 @@
 # Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q6_K, MTP MoE,
 # Dankserver, RTX 3090).
 #
-# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen
+# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp, built
-# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit
+# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
-# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS
+# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
-# Qwen3.6-35B-A3B model on 2026-08-24. Switched from CRACK Q8_0 to Q6_K on
+# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
-# 2026-08-25.
+# KV cache quantized to q8_KV with Hadamard transforms on both K and V.
+# --split-mode graph enabled (single GPU; retained per operator preference).
+#
+# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
+# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
+# so the q8_KV + Hadamard KV cache combos compile into the CUDA/CPU FA
+# kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24.
+# Switched from CRACK Q8_0 to Q6_K on 2026-08-25.
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
 # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
@@ -16,8 +23,12 @@
 MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
 MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf
 MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
-# Upstream llama.cpp binary (b10121).
+# ik_llama.cpp llama-server (built 2026-08-26 from ikawrakow/ik_llama.cpp @
-LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
+# b166e269; CUDA 13.2, -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_FA_ALL_QUANTS=ON,
+# arch 86 for the RTX 3090). Uses the staged pip cu13 toolkit at
+# /home/dank/cuda13-stage. Rollback: switch to
+# /home/dank/llama.cpp/build/bin/llama-server (upstream b10121).
+LLAMA_SERVER=/home/dank/ik_llama.cpp/build/bin/llama-server
 
 # Qwen-Sharp chat template from peculiar-ragdoll (tracked in the repo,
 # staged on the host).
@@ -35,10 +46,13 @@ CTX_SIZE=491520
 PARALLEL=3
 
 # Speculative decoding: ngram-mod only. MTP is intentionally disabled.
-SPEC_TYPE=ngram-mod
+# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
-SPEC_NGRAM_MOD_N_MIN=48
+# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
-SPEC_NGRAM_MOD_N_MAX=64
+#   old --spec-ngram-mod-n-min  (48) -> n_min
-SPEC_NGRAM_MOD_N_MATCH=24
+#   old --spec-ngram-mod-n-max  (64) -> n_max
+#   old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
+# SPEC_TYPE is passed through to --spec-type.
+SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
 
 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
 # layer audit found 630 MiB of expert tensors per block. Keep every
@@ -48,11 +62,23 @@ SPEC_NGRAM_MOD_N_MATCH=24
 # CPU buft name on this host.
 OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
-N_GPU_LAYERS=all
+# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
+# mean "load as many layers as possible". Expert exps are pinned to CPU by
+# OVERRIDE_EXPERT below.
+N_GPU_LAYERS=999
 
-# KV cache on GPU, both K and V at f16.
+# KV cache on GPU, both K and V at q8_KV (fast quant type on ik_llama, PR 208)
-CACHE_TYPE_K=f16
+# plus Hadamard transforms on both caches to preserve quality at 8-bit.
-CACHE_TYPE_V=f16
+# For quantized KV below Q6_0 the docs recommend --k-cache-hadamard; we apply
+# --v-cache-hadamard to V as well.
+CACHE_TYPE_K=q8_KV
+CACHE_TYPE_V=q8_KV
+K_CACHE_HADAMARD=1
+V_CACHE_HADAMARD=1
+
+# Split mode: graph (multi-GPU graph split; on a single 3090 this is a no-op
+# for correctness but kept per operator preference).
+SPLIT_MODE=graph
 
 VERBOSITY=4
 # Store context checkpoints in RAM to hold peak GPU memory down.

scripts/serve-frank-small.sh

@@ -1,11 +1,19 @@
 #!/usr/bin/env bash
 # Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as
-# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE
+# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE
 # experts on CPU).
 #
-# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
+# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
-# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this
+# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
-# service uses ngram-mod instead of MTP speculation.
+# this service uses ngram-mod instead of MTP speculation.
+#
+# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
+# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
+#
+# Split mode graph + --override-tensor CPU expert offload. Per ik_llama
+# README, `--split-mode graph` + partial GPU offload can occasionally produce
+# incoherent output; the documented fallback is `-cuda graphs=0`. Smoke-test
+# after each model load.
 #
 # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
 # --override-tensor pattern are pinned to CPU. Selected full-attention and
@@ -38,19 +46,19 @@ fi
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=491520}"
 : "${PARALLEL:=3}"
-: "${SPEC_TYPE:=ngram-mod}"
+: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
-: "${SPEC_NGRAM_MOD_N_MIN:=48}"
-: "${SPEC_NGRAM_MOD_N_MAX:=64}"
-: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
+: "${K_CACHE_HADAMARD:=0}"
+: "${V_CACHE_HADAMARD:=0}"
+: "${SPLIT_MODE:=none}"
 : "${VERBOSITY:=4}"
 # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
 # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
 # MoE expert weights and leave every other tensor on the GPU.
 : "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
 : "${OVERRIDE_BUFT:=CPU}"
-: "${N_GPU_LAYERS:=all}"
+: "${N_GPU_LAYERS:=999}"
 : "${BATCH_SIZE:=4096}"
 : "${UBATCH_SIZE:=1024}"
 : "${CTX_CHECKPOINTS:=16}"
@@ -81,12 +89,12 @@ exec "$LLAMA_SERVER" \
     --mmproj "$MMPROJ" \
     --chat-template-file "$CHAT_TEMPLATE_FILE" \
     --spec-type "$SPEC_TYPE" \
-    --spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \
-    --spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \
-    --spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \
     --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
     --cache-type-k "$CACHE_TYPE_K" \
     --cache-type-v "$CACHE_TYPE_V" \
+    $([ "$K_CACHE_HADAMARD" = "1" ] && printf -- "--k-cache-hadamard") \
+    $([ "$V_CACHE_HADAMARD" = "1" ] && printf -- "--v-cache-hadamard") \
+    --split-mode "$SPLIT_MODE" \
     --verbosity "$VERBOSITY" \
     --ctx-size "$CTX_SIZE" \
     --parallel "$PARALLEL" \

tests/frank_small_config.bats

@@ -35,17 +35,22 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
     [[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
     [[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
     [[ "$OVERRIDE_BUFT" == CPU ]]
-    [[ "$N_GPU_LAYERS" == all ]]
+    # ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
+    # loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
+    [[ "$N_GPU_LAYERS" == 999 ]]
 }
 
 @test "frank-default-small uses ngram-mod without MTP" {
-    grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE"
+    # ik_llama main uses the canonical --spec-type ngram-mod:<k=v> form; the
-    grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE"
+    # legacy --spec-ngram-mod-n-* flags are rejected.
-    grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE"
+    grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE"
-    grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE"
+    ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
+    ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
+    ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
     ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
     ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
-    grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE"
+    grep -q -- '--spec-type' "$SERVE_FILE"
+    ! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE"
 }
 
 @test "frank-default-small uses ubatch 1024" {