dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard

Browse Source 
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
This commit is contained in:
Frank
2026-08-26 05:43:35 -07:00
parent 8c41c821c4
commit 04a8fb7806
5 changed files with 84 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+6 -2
View File
@@ -189,7 +189,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw).
2. Run the bench on a scratch port (never the LLM service port 7770).
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
Scratch ports are not used for tests or benches on the LLM hosts — a
test process cannot share the single GPU with the running service anyway
(the model + KV fill the card), so changes are validated directly against
the service itself.
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
@@ -406,7 +410,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |
docs/LLM/ikllama-ornith-dankserver.md
+26 -21
View File
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, q6_0 KV cache with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 160k single slot, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K q8_0 / V q5_0 with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 160k ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -23,7 +23,7 @@ timestamp: 2026-08-26
The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
encoder). Served as a **single 160000-context slot** (`PARALLEL=1`).
encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
model runbook.
@@ -105,14 +105,16 @@ Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
`scripts/frank-default-small.service` in the repo. The service just calls
the serve script; the env file holds all tunables.
### KV cache: q8_0 → q6_0 + Hadamard
### KV cache: K q8_0 / V q5_0 + Hadamard
- `CACHE_TYPE_K=q6_0`, `CACHE_TYPE_V=q6_0` — 6-bit k-quant KV cache
(passes `--cache-type-k q6_0 --cache-type-v q6_0`).
- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q5_0` — asymmetric KV precision: the
K cache at 8-bit, the V cache at 5-bit (passes `--cache-type-k q8_0
--cache-type-v q5_0`). Keeping K higher-precision than V is a common
asymmetric budget split.
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
`--v-cache-hadamard`. The Hadamard transform preserves KV quality at
6-bit. Implemented for both CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK)
paths. Flash-attention is **on by default**.
quantized precision. Implemented for both CUDA (`ggml-cuda/hadamard.cu`)
and CPU (IQK) paths. Flash-attention is **on by default**.
> **KV cache type history on this build (`b166e269`):**
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
@@ -122,10 +124,12 @@ the serve script; the env file holds all tunables.
> - `q8_KV` **crashes** with a `ggml view` size assert
> (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
> QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
> - **`q6_0` + Hadamard works** and is what this deployment uses.
> - `q6_0` (both caches) boots and served, but with `PARALLEL=3` the KV
> tripled, so the K cache went back up to `q8_0` and V down to `q5_0` for
> the chosen K/V budget split.
At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving
~2.2 GiB headroom on the 24 GiB card after the GPU-side experts.
With 3 parallel slots at 160k context, the KV cache is ~3x a single slot's.
The GPU-side expert set is reduced to fit (see the expert section).
### Split mode: none (single GPU)
@@ -148,15 +152,17 @@ At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving
The expert-offload pattern from upstream llama.cpp carries over: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA
(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the
GPU headroom for lower latency; the rest of that CPU range stays on CPU.
the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
no exps and are untouched by the regex.
- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose
- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention +
35-40 set) are left off the list so their experts load to CUDA.
24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
full-attention set) are left off the list so their experts load to CUDA.
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
integer (the literal `all` is rejected). 999 means "load as many tensor
@@ -169,10 +175,9 @@ GPU headroom for lower latency; the rest of that CPU range stays on CPU.
> boot log for the running service.
To move more experts back to the GPU later, drop more block numbers from
`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom
first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the
free headroom was ~1.6 GiB, which is tighter than ideal — add more only
with a headroom check.
`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
cache there is little headroom, so the default keeps only 35-40 on GPU.
### Qwen-Sharp chat template via --jinja
scripts/frank-default-small.env
+26 -25
View File
@@ -5,8 +5,8 @@
# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
# KV cache quantized to iq5_k with Hadamard transforms on both K and V.
# Split mode none (single GPU; graph would require 2+ devices).
# KV cache K q8_0 / V q5_0 with Hadamard transforms on both (see the KV
# section below). Split mode none (single GPU; graph would require 2+).
#
# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
@@ -43,40 +43,41 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative
# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE
# =none) unless a user asks to re-enable ngram-mod/MTP.
# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama
# limits speculative decoding to a single slot, so SPEC stays none (MTP is
# also disabled). The 3 slots triple the KV cache, so enough expert weights
# are CPU-pinned for the cache to fit on the 24 GB card.
CTX_SIZE=160000
PARALLEL=1
PARALLEL=3
# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays
# off for the low-parameter MoE (3B active of 35B). MTP is also disabled.
# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at
# PARALLEL=3 a configured spec stage hard-exits at load. The low-parameter
# MoE (3B active of 35B) does not need it anyway. MTP is also disabled.
SPEC_TYPE=none
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep every
# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
# CPU buft name on this host.
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q8_0
# audit found ~1088 MiB of expert tensors per block (4 tensor × 272 MiB). Keep
# every final/output-adjacent block 35-40 on CUDA; full-attention blocks 3,7,
# 11,15,19,23 have no exps and are untouched by this regex. All other expert
# tensors are pinned to CPU. Buffer type "CPU" is the CPU buft on this host.
#
# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB)
# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU
# range below those stays on CPU.
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
# 2026-08-26: PARALLEL=3 (CTX 160000) triples the KV cache, so the blocks
# that were moved back to CUDA for fused-MoE (27-34) are pinned to CPU again
# to free GPU. Blocks 35-40 stay on CUDA. If it boot-loops, move the next
# lower block into the CPU regex one at a time.
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
# mean "load as many layers as possible". Expert exps are pinned to CPU by
# OVERRIDE_EXPERT below.
N_GPU_LAYERS=999
# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both
# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build
# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we
# verified works. Flash-attention is on by default.
CACHE_TYPE_K=q6_0
CACHE_TYPE_V=q6_0
# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with
# Hadamard transforms. The V cache at lower precision than K is a common
# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c
# :5427). Flash-attention is on by default.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q5_0
K_CACHE_HADAMARD=1
V_CACHE_HADAMARD=1
scripts/serve-frank-small.sh
+8 -7
View File
@@ -5,11 +5,12 @@
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding
# is disabled (ik_llama limits spec to a single slot).
#
# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build
# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env
# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.#
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
@@ -49,10 +50,10 @@ fi
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=160000}"
: "${PARALLEL:=1}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=q6_0}"
: "${CACHE_TYPE_V:=q6_0}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q5_0}"
: "${K_CACHE_HADAMARD:=0}"
: "${V_CACHE_HADAMARD:=0}"
: "${SPLIT_MODE:=none}"
tests/frank_small_config.bats
+18 -13
View File
@@ -14,41 +14,46 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE"
}
@test "frank-default-small uses a single 160000 context slot" {
@test "frank-default-small uses three 160000 context slots" {
grep -q '^CTX_SIZE=160000$' "$ENV_FILE"
grep -q '^PARALLEL=1$' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
}
@test "frank-default-small uses q6_0 KV cache with Hadamard" {
grep -q '^CACHE_TYPE_K=q6_0$' "$ENV_FILE"
grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE"
@test "frank-default-small uses K q8_0 + V q5_0 KV cache with Hadamard" {
grep -q '^CACHE_TYPE_K=q8_0$' "$ENV_FILE"
grep -q '^CACHE_TYPE_V=q5_0$' "$ENV_FILE"
grep -q '^K_CACHE_HADAMARD=1$' "$ENV_FILE"
grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE"
}
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
source "$ENV_FILE"
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
# output-adjacent final blocks stay on CUDA
[[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|37|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|38|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|39|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
# full-attention blocks (3,7,11,15,19,23) are untouched (no exps)
[[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|27|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|31|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
[[ "$OVERRIDE_BUFT" == CPU ]]
# ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
# loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
[[ "$N_GPU_LAYERS" == 999 ]]
}
@test "frank-default-small disables speculative decoding (single slot)" {
@test "frank-default-small disables speculative decoding (3 parallel slots)" {
# ik_llama limits speculative decoding to -np 1; with PARALLEL=3 a
# configured spec stage hard-exits, so SPEC_TYPE=none.
grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
grep -q '^PARALLEL=1$' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"