diff --git a/docs/LLM/ikllama-ornith-dankserver.md b/docs/LLM/ikllama-ornith-dankserver.md new file mode 100644 index 0000000..22b8d67 --- /dev/null +++ b/docs/LLM/ikllama-ornith-dankserver.md @@ -0,0 +1,208 @@ +--- +type: reference +title: ik_llama.cpp on Dankserver (Ornith Q6_K) +description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode graph, ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +tags: + - ikllama + - llamacpp + - cuda + - dankserver + - moe + - ornith + - kv-cache + - hadamard +timestamp: 2026-08-26 +--- + +# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK) + +> **Active 2026-08-26.** `frank-default-small` switched from upstream +> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the +> RTX 3090. This page documents the build, the mandatory FA build flags, and +> the MoE / CPU-offload runtime arguments. + +The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK +Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF. +Speculation stays ngram-only (MTP intentionally disabled). See +[[dealign-ornith-crack]] for the model runbook. + +## Why ik_llama.cpp + +ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA +optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV +cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is +exactly the supported hybrid the fork targets. Only CPU and CUDA backends +are fully functional; ROCm / Vulkan / Metal are unsupported upstream. + +## Build recipe (CUDA 13.2 via staged pip toolkit) + +The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit` +is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA +13.2 toolkit is installed via **pip** inside the vllm venv at +`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged, +self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned +`.so` aliases and static libs CMake needs. + +```bash +# 1. Clone a pinned ik_llama.cpp +git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp +cd /home/dank/ik_llama.cpp +git checkout b166e269 # the version this deployment was built from + +# 2. Configure (stage already symlinks nvcc/include/lib for cu13) +cmake -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DGGML_CUDA=ON \ + -DGGML_CUDA_FA_ALL_QUANTS=ON \ + -DGGML_IQK_FA_ALL_QUANTS=ON \ + -DCMAKE_CUDA_ARCHITECTURES=86 \ + -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \ + -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \ + -DGGML_NATIVE=ON + +# 3. Build +cmake --build build --config Release -j 8 +``` + +### ⚠️ MANDATORY: always include the FA all-quants flags + +> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache +> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with +> BOTH: +> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo +> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships +> a subset and **errors at runtime** telling you to recompile when a combo +> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on +> the GPU, so this is the flag that matters here. +> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/ +> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK** +> FA kernels. Relevant because the MoE experts run on the CPU; the KV +> cache that backs CPU-side attention uses the IQK FA path. +> +> These are the old "asymmetrical quants" compile-time gates. On a CUDA +> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a +> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time. + +`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so +`-march=native` enables the AVX512 / VNNI IQK kernels automatically — +verifiable with `system_info: AVX512_VNNI = 1` in the boot log). + +### CUDA stage note + +`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's +(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so` +and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the +CCCL CUDA version skew by defining +`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip +package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips +a strict header lint). Do not re-run configure without that flag or you get +a hard compile error. + +## Runtime configuration + +Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`, +`scripts/frank-default-small.service` in the repo. The service just calls +the serve script; the env file holds all tunables. + +### KV cache: q8_KV + Hadamard + +- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik + PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`. +- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and + `--v-cache-hadamard`. The Hadamard transform preserves KV quality at + 8-bit (docs recommend it for quantized KV below Q6_0). Implemented for + both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. + +These keep the 163840-slot / 491520-pool context while roughly halving KV +cache RAM vs f16. + +### Split-mode graph + +- `SPLIT_MODE=graph` → `--split-mode graph`. On a single 3090 this is a + no-op for correctness but keeps the graph scheduled across the compute + buffers. ik recommends it for MoE when more than one device is present; + here it is retained per operator preference alongside the explicit + `--override-tensor` expert pins. + +> **Caveat:** the ik_llama README warns that `--split-mode graph` combined +> with partial GPU offload (`--cpu-moe` / `--n-cpu-moe` / `--override-tensor`) +> can rarely produce incoherent output. The documented mitigation is +> `-cuda graphs=0`. Smoke-test after every load; if a response is gibberish, +> add `-cuda graphs=0` to the serve line and re-test. + +### Expert offload (MoE on CPU) + +The expert-offload pattern is unchanged from upstream llama.cpp: the +`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to +the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. + +- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch). +- `OVERRIDE_BUFT=CPU`. +- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an + integer (the literal `all` is rejected). 999 means "load as many tensor + layers as possible", while the expert override still pins the exps to CPU. + +### Speculative decoding: canonical ngram-mod + +ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match` +flags. The canonical form is a repeated `--spec-type ngram-mod:`: +- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24` + +Mapping from the old upstream config: +| legacy flag | old value | new canonical key | +|-------------|----------|-------------------| +| `--spec-ngram-mod-n-min` | 48 | `n_min=48` | +| `--spec-ngram-mod-n-max` | 64 | `n_max=64` | +| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` | + +MTP remains intentionally disabled. + +### Flash-attention + sampling + +Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no +`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and +checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread, +`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`, +`NO_MMAP`) are unchanged and pass straight through. + +## MoE / hybrid-arg reference (not in upstream llama.cpp) + +Several ik_llama knobs matter for a CPU-MoE offload host: + +| flag | meaning | our use | +|------|---------|---------| +| `--split-mode graph` | split tensors+graph across devices; strong for MoE | retained (single-GPU no-op) | +| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) | +| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) | +| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) | +| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained | +| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` | + +The current deployment uses `--override-tensor` (not `--fit` or +`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not +apply. + +## Rollback + +The previous runtime `upstream llama.cpp b10121` and its binary +(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To +rollback to the old stack, change `LLAMA_SERVER=` in +`scripts/frank-default-small.env` back to that path and +`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the +`--spec-ngram-mod-*` flags. The model file is not changed either way. + +## See Also + +- [[dealign-ornith-crack]] — the Ornith Q6_K model run +- [[Dankserver System Specs]] — the host +- [[deployment]] — service-scoped git deploy + +## Citations + +- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) — + MoE offload guidance and the `--split-mode graph` + CPU-offload warning +- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) — + build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS` +- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) — + q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and + `--split-mode graph` MoE note (PR 1080) \ No newline at end of file diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 6776ab0..e4e334f 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -1,7 +1,7 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small, +(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small, Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw (frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both services run the Qwen-Sharp chat template. @@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04] A[RTX 3090 24GB + 62GB RAM] - B[upstream llama.cpp b10121 + B2[ik_llama.cpp build b166e269 frank-default-small, port 7770] C[Ornith-1.5-35B-A3B-CRACK Q6_K -ngram-mod, selected experts offloaded via override, 491520 ctx] - A --> B --> C +ngram-mod, q8_KV + Hadamard cache, 491520 ctx] + A --> B2 --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] - D[gfx1151 iGPU 64 GiB unified memory] + E2[gfx1151 iGPU 64 GiB unified memory] E[charlie12345/ROCmFPX build-kairic (v81) frank-default-large, port 7770] F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT MTP draft + Turbo4 V cache, 262K ctx / 1 slot] - D --> E --> F + E2 --> E --> F end - I[LLM Clients / API] -->|HTTP :7770| B - I -->|HTTP :7770| E + CLI[LLM Clients / API] -->|HTTP :7770| B2 + CLI -->|HTTP :7770| E ``` ## Servers @@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot] on Dankclaw (retired) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) +- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for + frank-default-small (2026-08-26): q8_KV + Hadamard KV cache, + split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS + build flags - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment) -- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on - Dankserver (frank-default-small, active 2026-08-24) - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22) -- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small) +- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small) - [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large) ## CIRU Reference (Dankclaw) diff --git a/docs/index.md b/docs/index.md index cc015e7..673d534 100644 --- a/docs/index.md +++ b/docs/index.md @@ -110,9 +110,10 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) +- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_KV + Hadamard KV cache, split-mode graph; 2026-08-26) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21) -- [[Dankserver System Specs]] — the upstream llama.cpp box hardware +- [[Dankserver System Specs]] — the RTX 3090 box hardware - [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware - [[Dankclaw Improvements]] — host-level changes on the CIRU box diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index fd8d7c7..59697e5 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -1,11 +1,18 @@ # Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q6_K, MTP MoE, # Dankserver, RTX 3090). # -# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen -# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit -# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS -# Qwen3.6-35B-A3B model on 2026-08-24. Switched from CRACK Q8_0 to Q6_K on -# 2026-08-25. +# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp, built +# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB +# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights +# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe). +# KV cache quantized to q8_KV with Hadamard transforms on both K and V. +# --split-mode graph enabled (single GPU; retained per operator preference). +# +# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must +# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON +# so the q8_KV + Hadamard KV cache combos compile into the CUDA/CPU FA +# kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24. +# Switched from CRACK Q8_0 to Q6_K on 2026-08-25. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP @@ -16,8 +23,12 @@ MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf -# Upstream llama.cpp binary (b10121). -LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server +# ik_llama.cpp llama-server (built 2026-08-26 from ikawrakow/ik_llama.cpp @ +# b166e269; CUDA 13.2, -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_FA_ALL_QUANTS=ON, +# arch 86 for the RTX 3090). Uses the staged pip cu13 toolkit at +# /home/dank/cuda13-stage. Rollback: switch to +# /home/dank/llama.cpp/build/bin/llama-server (upstream b10121). +LLAMA_SERVER=/home/dank/ik_llama.cpp/build/bin/llama-server # Qwen-Sharp chat template from peculiar-ragdoll (tracked in the repo, # staged on the host). @@ -35,10 +46,13 @@ CTX_SIZE=491520 PARALLEL=3 # Speculative decoding: ngram-mod only. MTP is intentionally disabled. -SPEC_TYPE=ngram-mod -SPEC_NGRAM_MOD_N_MIN=48 -SPEC_NGRAM_MOD_N_MAX=64 -SPEC_NGRAM_MOD_N_MATCH=24 +# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy +# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config: +# old --spec-ngram-mod-n-min (48) -> n_min +# old --spec-ngram-mod-n-max (64) -> n_max +# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n +# SPEC_TYPE is passed through to --spec-type. +SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K # layer audit found 630 MiB of expert tensors per block. Keep every @@ -48,11 +62,23 @@ SPEC_NGRAM_MOD_N_MATCH=24 # CPU buft name on this host. OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU -N_GPU_LAYERS=all +# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to +# mean "load as many layers as possible". Expert exps are pinned to CPU by +# OVERRIDE_EXPERT below. +N_GPU_LAYERS=999 -# KV cache on GPU, both K and V at f16. -CACHE_TYPE_K=f16 -CACHE_TYPE_V=f16 +# KV cache on GPU, both K and V at q8_KV (fast quant type on ik_llama, PR 208) +# plus Hadamard transforms on both caches to preserve quality at 8-bit. +# For quantized KV below Q6_0 the docs recommend --k-cache-hadamard; we apply +# --v-cache-hadamard to V as well. +CACHE_TYPE_K=q8_KV +CACHE_TYPE_V=q8_KV +K_CACHE_HADAMARD=1 +V_CACHE_HADAMARD=1 + +# Split mode: graph (multi-GPU graph split; on a single 3090 this is a no-op +# for correctness but kept per operator preference). +SPLIT_MODE=graph VERBOSITY=4 # Store context checkpoints in RAM to hold peak GPU memory down. diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 154f127..a51fb33 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,11 +1,19 @@ #!/usr/bin/env bash # Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as -# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE +# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE # experts on CPU). # -# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total / -# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this -# service uses ngram-mod instead of MTP speculation. +# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a +# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; +# this service uses ngram-mod instead of MTP speculation. +# +# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with +# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file). +# +# Split mode graph + --override-tensor CPU expert offload. Per ik_llama +# README, `--split-mode graph` + partial GPU offload can occasionally produce +# incoherent output; the documented fallback is `-cuda graphs=0`. Smoke-test +# after each model load. # # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded # --override-tensor pattern are pinned to CPU. Selected full-attention and @@ -38,19 +46,19 @@ fi : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=491520}" : "${PARALLEL:=3}" -: "${SPEC_TYPE:=ngram-mod}" -: "${SPEC_NGRAM_MOD_N_MIN:=48}" -: "${SPEC_NGRAM_MOD_N_MAX:=64}" -: "${SPEC_NGRAM_MOD_N_MATCH:=24}" +: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" +: "${K_CACHE_HADAMARD:=0}" +: "${V_CACHE_HADAMARD:=0}" +: "${SPLIT_MODE:=none}" : "${VERBOSITY:=4}" # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the # MoE expert weights and leave every other tensor on the GPU. : "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}" : "${OVERRIDE_BUFT:=CPU}" -: "${N_GPU_LAYERS:=all}" +: "${N_GPU_LAYERS:=999}" : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=1024}" : "${CTX_CHECKPOINTS:=16}" @@ -81,12 +89,12 @@ exec "$LLAMA_SERVER" \ --mmproj "$MMPROJ" \ --chat-template-file "$CHAT_TEMPLATE_FILE" \ --spec-type "$SPEC_TYPE" \ - --spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \ - --spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \ - --spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \ --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \ --cache-type-k "$CACHE_TYPE_K" \ --cache-type-v "$CACHE_TYPE_V" \ + $([ "$K_CACHE_HADAMARD" = "1" ] && printf -- "--k-cache-hadamard") \ + $([ "$V_CACHE_HADAMARD" = "1" ] && printf -- "--v-cache-hadamard") \ + --split-mode "$SPLIT_MODE" \ --verbosity "$VERBOSITY" \ --ctx-size "$CTX_SIZE" \ --parallel "$PARALLEL" \ diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index bb6c52c..ed0862e 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -35,17 +35,22 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" [[ "$OVERRIDE_EXPERT" != *'|36|'* ]] [[ "$OVERRIDE_EXPERT" != *'|40|'* ]] [[ "$OVERRIDE_BUFT" == CPU ]] - [[ "$N_GPU_LAYERS" == all ]] + # ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999 + # loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU. + [[ "$N_GPU_LAYERS" == 999 ]] } @test "frank-default-small uses ngram-mod without MTP" { - grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE" - grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE" - grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE" - grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE" + # ik_llama main uses the canonical --spec-type ngram-mod: form; the + # legacy --spec-ngram-mod-n-* flags are rejected. + grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE" + ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE" + ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE" + ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE" ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE" ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE" - grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE" + grep -q -- '--spec-type' "$SERVE_FILE" + ! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE" } @test "frank-default-small uses ubatch 1024" {