frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13 toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the old binary is kept for rollback. Runtime changes: - KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard - --split-mode graph (single-GPU no-op, retained per operator pref) - migrate ngram speculation to canonical --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24 (legacy --spec-ngram-mod-* are rejected in ik main) - N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay pinned to CPU via --override-tensor Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory for the quantized KV cache; recipe + reasoning baked into docs/LLM/ikllama-ornith.md. Tests and both indices updated.
This commit is contained in:
Frank
@@ -0,0 +1,208 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_KV KV cache with Hadamard, split-mode graph, ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- kv-cache
|
||||
- hadamard
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
||||
|
||||
> **Active 2026-08-26.** `frank-default-small` switched from upstream
|
||||
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
||||
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
||||
> the MoE / CPU-offload runtime arguments.
|
||||
|
||||
The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
|
||||
Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
|
||||
Speculation stays ngram-only (MTP intentionally disabled). See
|
||||
[[dealign-ornith-crack]] for the model runbook.
|
||||
|
||||
## Why ik_llama.cpp
|
||||
|
||||
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
|
||||
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
|
||||
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
|
||||
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
|
||||
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
|
||||
|
||||
## Build recipe (CUDA 13.2 via staged pip toolkit)
|
||||
|
||||
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
|
||||
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
|
||||
13.2 toolkit is installed via **pip** inside the vllm venv at
|
||||
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
|
||||
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
|
||||
`.so` aliases and static libs CMake needs.
|
||||
|
||||
```bash
|
||||
# 1. Clone a pinned ik_llama.cpp
|
||||
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
|
||||
cd /home/dank/ik_llama.cpp
|
||||
git checkout b166e269 # the version this deployment was built from
|
||||
|
||||
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
|
||||
cmake -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_FA_ALL_QUANTS=ON \
|
||||
-DGGML_IQK_FA_ALL_QUANTS=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 \
|
||||
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
|
||||
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
|
||||
-DGGML_NATIVE=ON
|
||||
|
||||
# 3. Build
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
### ⚠️ MANDATORY: always include the FA all-quants flags
|
||||
|
||||
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
|
||||
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
|
||||
> BOTH:
|
||||
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
|
||||
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
|
||||
> a subset and **errors at runtime** telling you to recompile when a combo
|
||||
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
|
||||
> the GPU, so this is the flag that matters here.
|
||||
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
|
||||
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
|
||||
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
|
||||
> cache that backs CPU-side attention uses the IQK FA path.
|
||||
>
|
||||
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
|
||||
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
|
||||
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
|
||||
|
||||
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
|
||||
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
|
||||
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
|
||||
|
||||
### CUDA stage note
|
||||
|
||||
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
|
||||
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
|
||||
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
|
||||
CCCL CUDA version skew by defining
|
||||
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
|
||||
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
|
||||
a strict header lint). Do not re-run configure without that flag or you get
|
||||
a hard compile error.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
|
||||
`scripts/frank-default-small.service` in the repo. The service just calls
|
||||
the serve script; the env file holds all tunables.
|
||||
|
||||
### KV cache: q8_KV + Hadamard
|
||||
|
||||
- `CACHE_TYPE_K=q8_KV`, `CACHE_TYPE_V=q8_KV` — fast quant type (ik
|
||||
PR 208). Passes `--cache-type-k q8_KV --cache-type-v q8_KV`.
|
||||
- `K_CACHE_HADAMARD=1`, `V_CACHE_HADAMARD=1` — `--k-cache-hadamard` and
|
||||
`--v-cache-hadamard`. The Hadamard transform preserves KV quality at
|
||||
8-bit (docs recommend it for quantized KV below Q6_0). Implemented for
|
||||
both the CUDA (`ggml-cuda/hadamard.cu`) and CPU (IQK) paths.
|
||||
|
||||
These keep the 163840-slot / 491520-pool context while roughly halving KV
|
||||
cache RAM vs f16.
|
||||
|
||||
### Split-mode graph
|
||||
|
||||
- `SPLIT_MODE=graph` → `--split-mode graph`. On a single 3090 this is a
|
||||
no-op for correctness but keeps the graph scheduled across the compute
|
||||
buffers. ik recommends it for MoE when more than one device is present;
|
||||
here it is retained per operator preference alongside the explicit
|
||||
`--override-tensor` expert pins.
|
||||
|
||||
> **Caveat:** the ik_llama README warns that `--split-mode graph` combined
|
||||
> with partial GPU offload (`--cpu-moe` / `--n-cpu-moe` / `--override-tensor`)
|
||||
> can rarely produce incoherent output. The documented mitigation is
|
||||
> `-cuda graphs=0`. Smoke-test after every load; if a response is gibberish,
|
||||
> add `-cuda graphs=0` to the serve line and re-test.
|
||||
|
||||
### Expert offload (MoE on CPU)
|
||||
|
||||
The expert-offload pattern is unchanged from upstream llama.cpp: the
|
||||
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
||||
the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
|
||||
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex (same as before the switch).
|
||||
- `OVERRIDE_BUFT=CPU`.
|
||||
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
||||
integer (the literal `all` is rejected). 999 means "load as many tensor
|
||||
layers as possible", while the expert override still pins the exps to CPU.
|
||||
|
||||
### Speculative decoding: canonical ngram-mod
|
||||
|
||||
ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
|
||||
flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
|
||||
- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`
|
||||
|
||||
Mapping from the old upstream config:
|
||||
| legacy flag | old value | new canonical key |
|
||||
|-------------|----------|-------------------|
|
||||
| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
|
||||
| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
|
||||
| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |
|
||||
|
||||
MTP remains intentionally disabled.
|
||||
|
||||
### Flash-attention + sampling
|
||||
|
||||
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
|
||||
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
|
||||
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
|
||||
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
|
||||
`NO_MMAP`) are unchanged and pass straight through.
|
||||
|
||||
## MoE / hybrid-arg reference (not in upstream llama.cpp)
|
||||
|
||||
Several ik_llama knobs matter for a CPU-MoE offload host:
|
||||
|
||||
| flag | meaning | our use |
|
||||
|------|---------|---------|
|
||||
| `--split-mode graph` | split tensors+graph across devices; strong for MoE | retained (single-GPU no-op) |
|
||||
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
|
||||
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
|
||||
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
|
||||
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
|
||||
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
|
||||
|
||||
The current deployment uses `--override-tensor` (not `--fit` or
|
||||
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
|
||||
apply.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous runtime `upstream llama.cpp b10121` and its binary
|
||||
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
||||
rollback to the old stack, change `LLAMA_SERVER=` in
|
||||
`scripts/frank-default-small.env` back to that path and
|
||||
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
|
||||
`--spec-ngram-mod-*` flags. The model file is not changed either way.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
|
||||
- [[Dankserver System Specs]] — the host
|
||||
- [[deployment]] — service-scoped git deploy
|
||||
|
||||
## Citations
|
||||
|
||||
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
|
||||
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
|
||||
- [ik_llama.cpp docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
|
||||
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
|
||||
- [ik_llama.cpp docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
|
||||
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
|
||||
`--split-mode graph` MoE note (PR 1080)
|
||||
+13
-11
@@ -1,7 +1,7 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
@@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template.
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[upstream llama.cpp b10121
|
||||
B2[ik_llama.cpp build b166e269
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
ngram-mod, selected experts offloaded via override, 491520 ctx]
|
||||
A --> B --> C
|
||||
ngram-mod, q8_KV + Hadamard cache, 491520 ctx]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
I[LLM Clients / API] -->|HTTP :7770| B
|
||||
I -->|HTTP :7770| E
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
@@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_KV + Hadamard KV cache,
|
||||
split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS
|
||||
build flags
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
+2
-1
@@ -110,9 +110,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_KV + Hadamard KV cache, split-mode graph; 2026-08-26)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
||||
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
||||
|
||||
|
||||
@@ -1,11 +1,18 @@
|
||||
# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q6_K, MTP MoE,
|
||||
# Dankserver, RTX 3090).
|
||||
#
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). RTX 3090 24 GB; Ryzen
|
||||
# 7 7700X; 62 GB RAM. MoE expert weights pinned to CPU with an explicit
|
||||
# --override-tensor regex (not --n-cpu-moe). Replaced the HauhauCS
|
||||
# Qwen3.6-35B-A3B model on 2026-08-24. Switched from CRACK Q8_0 to Q6_K on
|
||||
# 2026-08-25.
|
||||
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp, built
|
||||
# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
|
||||
# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
|
||||
# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
|
||||
# KV cache quantized to q8_KV with Hadamard transforms on both K and V.
|
||||
# --split-mode graph enabled (single GPU; retained per operator preference).
|
||||
#
|
||||
# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
|
||||
# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
|
||||
# so the q8_KV + Hadamard KV cache combos compile into the CUDA/CPU FA
|
||||
# kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24.
|
||||
# Switched from CRACK Q8_0 to Q6_K on 2026-08-25.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
@@ -16,8 +23,12 @@
|
||||
MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
|
||||
MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf
|
||||
MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
|
||||
# Upstream llama.cpp binary (b10121).
|
||||
LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
|
||||
# ik_llama.cpp llama-server (built 2026-08-26 from ikawrakow/ik_llama.cpp @
|
||||
# b166e269; CUDA 13.2, -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_FA_ALL_QUANTS=ON,
|
||||
# arch 86 for the RTX 3090). Uses the staged pip cu13 toolkit at
|
||||
# /home/dank/cuda13-stage. Rollback: switch to
|
||||
# /home/dank/llama.cpp/build/bin/llama-server (upstream b10121).
|
||||
LLAMA_SERVER=/home/dank/ik_llama.cpp/build/bin/llama-server
|
||||
|
||||
# Qwen-Sharp chat template from peculiar-ragdoll (tracked in the repo,
|
||||
# staged on the host).
|
||||
@@ -35,10 +46,13 @@ CTX_SIZE=491520
|
||||
PARALLEL=3
|
||||
|
||||
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
|
||||
SPEC_TYPE=ngram-mod
|
||||
SPEC_NGRAM_MOD_N_MIN=48
|
||||
SPEC_NGRAM_MOD_N_MAX=64
|
||||
SPEC_NGRAM_MOD_N_MATCH=24
|
||||
# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
|
||||
# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
|
||||
# old --spec-ngram-mod-n-min (48) -> n_min
|
||||
# old --spec-ngram-mod-n-max (64) -> n_max
|
||||
# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
|
||||
# SPEC_TYPE is passed through to --spec-type.
|
||||
SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
|
||||
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
|
||||
# layer audit found 630 MiB of expert tensors per block. Keep every
|
||||
@@ -48,11 +62,23 @@ SPEC_NGRAM_MOD_N_MATCH=24
|
||||
# CPU buft name on this host.
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
N_GPU_LAYERS=all
|
||||
# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
|
||||
# mean "load as many layers as possible". Expert exps are pinned to CPU by
|
||||
# OVERRIDE_EXPERT below.
|
||||
N_GPU_LAYERS=999
|
||||
|
||||
# KV cache on GPU, both K and V at f16.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
# KV cache on GPU, both K and V at q8_KV (fast quant type on ik_llama, PR 208)
|
||||
# plus Hadamard transforms on both caches to preserve quality at 8-bit.
|
||||
# For quantized KV below Q6_0 the docs recommend --k-cache-hadamard; we apply
|
||||
# --v-cache-hadamard to V as well.
|
||||
CACHE_TYPE_K=q8_KV
|
||||
CACHE_TYPE_V=q8_KV
|
||||
K_CACHE_HADAMARD=1
|
||||
V_CACHE_HADAMARD=1
|
||||
|
||||
# Split mode: graph (multi-GPU graph split; on a single 3090 this is a no-op
|
||||
# for correctness but kept per operator preference).
|
||||
SPLIT_MODE=graph
|
||||
|
||||
VERBOSITY=4
|
||||
# Store context checkpoints in RAM to hold peak GPU memory down.
|
||||
|
||||
@@ -1,11 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as
|
||||
# frank-default-small via upstream llama.cpp on Dankserver (RTX 3090, MoE
|
||||
# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE
|
||||
# experts on CPU).
|
||||
#
|
||||
# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
|
||||
# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this
|
||||
# service uses ngram-mod instead of MTP speculation.
|
||||
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
|
||||
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
|
||||
# this service uses ngram-mod instead of MTP speculation.
|
||||
#
|
||||
# KV cache: q8_KV with Hadamard transforms on K and V (requires a build with
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
|
||||
#
|
||||
# Split mode graph + --override-tensor CPU expert offload. Per ik_llama
|
||||
# README, `--split-mode graph` + partial GPU offload can occasionally produce
|
||||
# incoherent output; the documented fallback is `-cuda graphs=0`. Smoke-test
|
||||
# after each model load.
|
||||
#
|
||||
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
|
||||
# --override-tensor pattern are pinned to CPU. Selected full-attention and
|
||||
@@ -38,19 +46,19 @@ fi
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=491520}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${SPEC_TYPE:=ngram-mod}"
|
||||
: "${SPEC_NGRAM_MOD_N_MIN:=48}"
|
||||
: "${SPEC_NGRAM_MOD_N_MAX:=64}"
|
||||
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${K_CACHE_HADAMARD:=0}"
|
||||
: "${V_CACHE_HADAMARD:=0}"
|
||||
: "${SPLIT_MODE:=none}"
|
||||
: "${VERBOSITY:=4}"
|
||||
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
|
||||
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
|
||||
# MoE expert weights and leave every other tensor on the GPU.
|
||||
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
|
||||
: "${OVERRIDE_BUFT:=CPU}"
|
||||
: "${N_GPU_LAYERS:=all}"
|
||||
: "${N_GPU_LAYERS:=999}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
: "${CTX_CHECKPOINTS:=16}"
|
||||
@@ -81,12 +89,12 @@ exec "$LLAMA_SERVER" \
|
||||
--mmproj "$MMPROJ" \
|
||||
--chat-template-file "$CHAT_TEMPLATE_FILE" \
|
||||
--spec-type "$SPEC_TYPE" \
|
||||
--spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \
|
||||
--spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \
|
||||
--spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \
|
||||
--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
$([ "$K_CACHE_HADAMARD" = "1" ] && printf -- "--k-cache-hadamard") \
|
||||
$([ "$V_CACHE_HADAMARD" = "1" ] && printf -- "--v-cache-hadamard") \
|
||||
--split-mode "$SPLIT_MODE" \
|
||||
--verbosity "$VERBOSITY" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
|
||||
@@ -35,17 +35,22 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
|
||||
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
|
||||
[[ "$OVERRIDE_BUFT" == CPU ]]
|
||||
[[ "$N_GPU_LAYERS" == all ]]
|
||||
# ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
|
||||
# loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
|
||||
[[ "$N_GPU_LAYERS" == 999 ]]
|
||||
}
|
||||
|
||||
@test "frank-default-small uses ngram-mod without MTP" {
|
||||
grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE"
|
||||
grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE"
|
||||
grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE"
|
||||
grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE"
|
||||
# ik_llama main uses the canonical --spec-type ngram-mod:<k=v> form; the
|
||||
# legacy --spec-ngram-mod-n-* flags are rejected.
|
||||
grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
|
||||
! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
|
||||
! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
|
||||
grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE"
|
||||
grep -q -- '--spec-type' "$SERVE_FILE"
|
||||
! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small uses ubatch 1024" {
|
||||
|
||||
Reference in New Issue
Block a user