diff --git a/AGENTS.md b/AGENTS.md index 808019b..4230a64 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -189,7 +189,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE 1. Stop the service before any benchmark: `sudo systemctl stop frank-default-small` (Dankserver) or `sudo systemctl stop frank-default-large` (Dankclaw). -2. Run the bench on a scratch port (never the LLM service port 7770). +2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM). + Scratch ports are not used for tests or benches on the LLM hosts — a + test process cannot share the single GPU with the running service anyway + (the model + KV fill the card), so changes are validated directly against + the service itself. 3. Restart the service when the bench completes: `sudo systemctl start `. 4. Verify health before reporting: `curl http://192.168.0.235:7770/health` or @@ -406,7 +410,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history. | Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. | | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | | Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). | -| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. | +| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | | Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. | diff --git a/docs/LLM/ikllama-ornith-dankserver.md b/docs/LLM/ikllama-ornith-dankserver.md index d84b58a..2de8590 100644 --- a/docs/LLM/ikllama-ornith-dankserver.md +++ b/docs/LLM/ikllama-ornith-dankserver.md @@ -1,7 +1,7 @@ --- type: reference title: ik_llama.cpp on Dankserver (Ornith Q8_0) -description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, q6_0 KV cache with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 160k single slot, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K q8_0 / V q5_0 with Hadamard (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 160k ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. tags: - ikllama - llamacpp @@ -23,7 +23,7 @@ timestamp: 2026-08-26 The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision -encoder). Served as a **single 160000-context slot** (`PARALLEL=1`). +encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`). Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the model runbook. @@ -105,14 +105,16 @@ Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`, `scripts/frank-default-small.service` in the repo. The service just calls the serve script; the env file holds all tunables. -### KV cache: q8_0 → q6_0 + Hadamard +### KV cache: K q8_0 / V q5_0 + Hadamard -- `CACHE_TYPE_K=q6_0`, `CACHE_TYPE_V=q6_0` — 6-bit k-quant KV cache - (passes `--cache-type-k q6_0 --cache-type-v q6_0`). +- `CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q5_0` — asymmetric KV precision: the + K cache at 8-bit, the V cache at 5-bit (passes `--cache-type-k q8_0 + --cache-type-v q5_0`). Keeping K higher-precision than V is a common + asymmetric budget split. - `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and `--v-cache-hadamard`. The Hadamard transform preserves KV quality at - 6-bit. Implemented for both CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) - paths. Flash-attention is **on by default**. + quantized precision. Implemented for both CUDA (`ggml-cuda/hadamard.cu`) + and CPU (IQK) paths. Flash-attention is **on by default**. > **KV cache type history on this build (`b166e269`):** > - `iq5_k` is **not** a KV-cache type — the cache whitelist is @@ -122,10 +124,12 @@ the serve script; the env file holds all tunables. > - `q8_KV` **crashes** with a `ggml view` size assert > (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this > QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation. -> - **`q6_0` + Hadamard works** and is what this deployment uses. +> - `q6_0` (both caches) boots and served, but with `PARALLEL=3` the KV +> tripled, so the K cache went back up to `q8_0` and V down to `q5_0` for +> the chosen K/V budget split. -At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving -~2.2 GiB headroom on the 24 GiB card after the GPU-side experts. +With 3 parallel slots at 160k context, the KV cache is ~3x a single slot's. +The GPU-side expert set is reduced to fit (see the expert section). ### Split mode: none (single GPU) @@ -148,15 +152,17 @@ At 160k single slot the q6_0 KV self is ~1270 MiB (K 634 + V 634), leaving The expert-offload pattern from upstream llama.cpp carries over: the `--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to -the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. -On 2026-08-26 we brought expert blocks **30, 32, 33, 34** back to CUDA -(~2.5 GiB), then **28 and 29** (~1.26 GiB) for the fused-MoE run, to use the -GPU headroom for lower latency; the rest of that CPU range stays on CPU. +the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the +Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088 +MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks +are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and +quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have +no exps and are untouched by the regex. -- `OVERRIDE_EXPERT` — the layer-bounded regex listing only the blocks whose +- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| - 24|25|26)`. Blocks 28,29,30,32,33,34 (and the always-on full-attention + - 35-40 set) are left off the list so their experts load to CUDA. + 24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on + full-attention set) are left off the list so their experts load to CUDA. - `OVERRIDE_BUFT=CPU`. - `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an integer (the literal `all` is rejected). 999 means "load as many tensor @@ -169,10 +175,9 @@ GPU headroom for lower latency; the rest of that CPU range stays on CPU. > boot log for the running service. To move more experts back to the GPU later, drop more block numbers from -`OVERRIDE_EXPERT` (each block is ~630 MiB; verify `nvidia-smi` headroom -first and keep ~2-3 GiB margin for KV/compute). After 28,29 were added the -free headroom was ~1.6 GiB, which is tighter than ideal — add more only -with a headroom check. +`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi` +headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV +cache there is little headroom, so the default keeps only 35-40 on GPU. ### Qwen-Sharp chat template via --jinja diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 38dfd77..3081d90 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -5,8 +5,8 @@ # 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB # RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights # pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe). -# KV cache quantized to iq5_k with Hadamard transforms on both K and V. -# Split mode none (single GPU; graph would require 2+ devices). +# KV cache K q8_0 / V q5_0 with Hadamard transforms on both (see the KV +# section below). Split mode none (single GPU; graph would require 2+). # # Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must # ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON @@ -43,40 +43,41 @@ ALIAS=frank-default-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative -# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE -# =none) unless a user asks to re-enable ngram-mod/MTP. +# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama +# limits speculative decoding to a single slot, so SPEC stays none (MTP is +# also disabled). The 3 slots triple the KV cache, so enough expert weights +# are CPU-pinned for the cache to fit on the 24 GB card. CTX_SIZE=160000 -PARALLEL=1 +PARALLEL=3 -# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays -# off for the low-parameter MoE (3B active of 35B). MTP is also disabled. +# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at +# PARALLEL=3 a configured spec stage hard-exits at load. The low-parameter +# MoE (3B active of 35B) does not need it anyway. MTP is also disabled. SPEC_TYPE=none -# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K -# layer audit found 630 MiB of expert tensors per block. Keep every -# full-attention block plus every block 35-40 on CUDA. The unique GPU set is -# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to -# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the -# CPU buft name on this host. +# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q8_0 +# audit found ~1088 MiB of expert tensors per block (4 tensor × 272 MiB). Keep +# every final/output-adjacent block 35-40 on CUDA; full-attention blocks 3,7, +# 11,15,19,23 have no exps and are untouched by this regex. All other expert +# tensors are pinned to CPU. Buffer type "CPU" is the CPU buft on this host. # -# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to -# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB) -# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU -# range below those stays on CPU. -OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +# 2026-08-26: PARALLEL=3 (CTX 160000) triples the KV cache, so the blocks +# that were moved back to CUDA for fused-MoE (27-34) are pinned to CPU again +# to free GPU. Blocks 35-40 stay on CUDA. If it boot-loops, move the next +# lower block into the CPU regex one at a time. +OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU # ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to # mean "load as many layers as possible". Expert exps are pinned to CPU by # OVERRIDE_EXPERT below. N_GPU_LAYERS=999 -# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both -# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build -# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we -# verified works. Flash-attention is on by default. -CACHE_TYPE_K=q6_0 -CACHE_TYPE_V=q6_0 +# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with +# Hadamard transforms. The V cache at lower precision than K is a common +# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c +# :5427). Flash-attention is on by default. +CACHE_TYPE_K=q8_0 +CACHE_TYPE_V=q5_0 K_CACHE_HADAMARD=1 V_CACHE_HADAMARD=1 diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 4362bf9..865f63d 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -5,11 +5,12 @@ # # Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a # 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF. -# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled. +# Three parallel slots (PARALLEL=3), each with 160000 ctx. Speculative decoding +# is disabled (ik_llama limits spec to a single slot). # -# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with -# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file). -# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.# +# KV cache: K q8_0 + V q5_0 with Hadamard transforms (requires a build +# with GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env +# file). q8_KV crashes at init on this arch (ggml.c:5427); q8_0/q5_0 work.# # Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects # split-mode graph with fewer than 2 devices (hard load error). @@ -49,10 +50,10 @@ fi : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=160000}" -: "${PARALLEL:=1}" +: "${PARALLEL:=3}" : "${SPEC_TYPE:=none}" -: "${CACHE_TYPE_K:=q6_0}" -: "${CACHE_TYPE_V:=q6_0}" +: "${CACHE_TYPE_K:=q8_0}" +: "${CACHE_TYPE_V:=q5_0}" : "${K_CACHE_HADAMARD:=0}" : "${V_CACHE_HADAMARD:=0}" : "${SPLIT_MODE:=none}" diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index 9f5e2aa..59f316d 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -14,41 +14,46 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE" } -@test "frank-default-small uses a single 160000 context slot" { +@test "frank-default-small uses three 160000 context slots" { grep -q '^CTX_SIZE=160000$' "$ENV_FILE" - grep -q '^PARALLEL=1$' "$ENV_FILE" + grep -q '^PARALLEL=3$' "$ENV_FILE" } -@test "frank-default-small uses q6_0 KV cache with Hadamard" { - grep -q '^CACHE_TYPE_K=q6_0$' "$ENV_FILE" - grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE" +@test "frank-default-small uses K q8_0 + V q5_0 KV cache with Hadamard" { + grep -q '^CACHE_TYPE_K=q8_0$' "$ENV_FILE" + grep -q '^CACHE_TYPE_V=q5_0$' "$ENV_FILE" grep -q '^K_CACHE_HADAMARD=1$' "$ENV_FILE" grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE" } @test "frank-default-small keeps full-attention and final blocks on CUDA" { source "$ENV_FILE" - [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]] + [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]] + # output-adjacent final blocks stay on CUDA + [[ "$OVERRIDE_EXPERT" != *'|35|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|36|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|37|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|38|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|39|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|40|'* ]] + # full-attention blocks (3,7,11,15,19,23) are untouched (no exps) [[ "$OVERRIDE_EXPERT" != *'|3|'* ]] [[ "$OVERRIDE_EXPERT" != *'|7|'* ]] [[ "$OVERRIDE_EXPERT" != *'|11|'* ]] [[ "$OVERRIDE_EXPERT" != *'|15|'* ]] [[ "$OVERRIDE_EXPERT" != *'|19|'* ]] [[ "$OVERRIDE_EXPERT" != *'|23|'* ]] - [[ "$OVERRIDE_EXPERT" != *'|27|'* ]] - [[ "$OVERRIDE_EXPERT" != *'|31|'* ]] - [[ "$OVERRIDE_EXPERT" != *'|35|'* ]] - [[ "$OVERRIDE_EXPERT" != *'|36|'* ]] - [[ "$OVERRIDE_EXPERT" != *'|40|'* ]] [[ "$OVERRIDE_BUFT" == CPU ]] # ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999 # loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU. [[ "$N_GPU_LAYERS" == 999 ]] } -@test "frank-default-small disables speculative decoding (single slot)" { +@test "frank-default-small disables speculative decoding (3 parallel slots)" { + # ik_llama limits speculative decoding to -np 1; with PARALLEL=3 a + # configured spec stage hard-exits, so SPEC_TYPE=none. grep -q '^SPEC_TYPE=none$' "$ENV_FILE" - grep -q '^PARALLEL=1$' "$ENV_FILE" + grep -q '^PARALLEL=3$' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"