Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

12 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)

Active 2026-08-26. frank-zealot switched from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (commit b166e269) on the RTX 3090. This page documents the build, the mandatory FA build flags, and the MoE / CPU-offload runtime arguments.

The model is dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q8_0), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision encoder). Served as three parallel 160000-context slots (PARALLEL=3). Speculative decoding is disabled. See dealign-ornith-crack for the model runbook.

Why ik_llama.cpp

ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA optimizations, IQK/AVX-512 kernels, --fit auto-offload, and fast q8_KV cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is exactly the supported hybrid the fork targets. Only CPU and CUDA backends are fully functional; ROCm / Vulkan / Metal are unsupported upstream.

Build recipe (CUDA 13.2 via staged pip toolkit)

The system CUDA toolkit on Dankserver is removed (nvidia-cuda-toolkit is in rc state; no /usr/local/cuda*, no /usr/bin/nvcc). The CUDA 13.2 toolkit is installed via pip inside the vllm venv at /home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/. A staged, self-contained toolchain at /home/dank/cuda13-stage adds the unversioned .so aliases and static libs CMake needs.

# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269   # the version this deployment was built from

# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_IQK_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
  -DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
  -DGGML_NATIVE=ON

# 3. Build
cmake --build build --config Release -j 8

⚠️ MANDATORY: always include the FA all-quants flags

Rule. Any ik_llama.cpp build that serves a quantized KV cache (q8_KV / q8_0 / Hadamard / any non-f16 KV) MUST be compiled with BOTH:

-DGGML_CUDA_FA_ALL_QUANTS=ON — compiles every K/V cache quant combo into the CUDA FlashAttention kernels. Without it, CUDA FA only ships a subset and errors at runtime telling you to recompile when a combo (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on the GPU, so this is the flag that matters here.

-DGGML_IQK_FA_ALL_QUANTS=ON — compiles all quant types (Q4_0/Q4_1/ IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the CPU/IQK FA kernels. Relevant because the MoE experts run on the CPU; the KV cache that backs CPU-side attention uses the IQK FA path.

These are the old "asymmetrical quants" compile-time gates. On a CUDA host, omitting GGML_CUDA_FA_ALL_QUANTS is the #1 way to get a "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.

GGML_NATIVE=ON lets GCC target the host (the Ryzen 7 7700X is Zen4, so -march=native enables the AVX512 / VNNI IQK kernels automatically — verifiable with system_info: AVX512_VNNI = 1 in the boot log).

CUDA stage note

/home/dank/cuda13-stage is the pip cu13 toolkit plus the symlinks CMake's (CUDAToolkit) finder needs (libcudart.so, libcublas.so, libcublasLt.so and static libcudadevrt.a/libcudart_static.a). It also works around the CCCL CUDA version skew by defining -DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK in CMAKE_CUDA_FLAGS (the pip package ships nvcc 13.2.78 but reports CUDART_VERSION 13000, which trips a strict header lint). Do not re-run configure without that flag or you get a hard compile error.

Runtime configuration

Files: scripts/serve-frank-zealot.sh, scripts/frank-zealot.env, scripts/frank-zealot.service in the repo. The service just calls the serve script; the env file holds all tunables.

KV cache: K default f16 / V q6_0 + Hadamard (V only)

CACHE_TYPE_K=f16 — K cache at the default full-precision f16 (passes --cache-type-k f16). No Hadamard on K.
CACHE_TYPE_V=q6_0 — V cache at 6-bit k-quant (passes --cache-type-v q6_0).
V_CACHE_HADAMARD=1 — --v-cache-hadamard on V only. The Hadamard transform preserves V quality at 6-bit; K is full-precision f16 and does not need it (K_CACHE_HADAMARD=0). Implemented for both CUDA (ggml-cuda/hadamard.cu) and CPU (IQK) paths. Flash-attention is on by default.

KV cache type history on this build (b166e269):

iq5_k is not a KV-cache type — the cache whitelist is f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV; iq5_* only exist as weight quants. --cache-type-k iq5_k throws Invalid cache type.

q8_KV crashes with a ggml view size assert (ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)) on this QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.

The deployment settled on K at default f16 (full precision, no Hadamard) with V at q6_0 + Hadamard.

With 3 parallel slots at 163840 context the KV cache is large; keeping K at full precision f16 while quantizing only V (with Hadamard) is the chosen budget split. The GPU-side expert set stays reduced to fit (see the expert section).

Split mode: none (single GPU)

SPLIT_MODE=none → --split-mode none (the default). Dankserver has a single RTX 3090 yet ik_llama rejects --split-mode graph on a box with a single device:

llama error loading model: It is not possible to use split mode 'graph' with less than 2 devices

So graph is not a harmless no-op on one GPU — it is a hard load error (this bit us on the first deploy and loop the service). It also still works with the --override-tensor expert pins.

Lesson. ik_llama --split-mode graph requires ≥2 devices. On a single-GPU host you must use none (or layer, which also needs >1). Do not assume --split-mode graph degrades gracefully on one GPU.

Expert offload (MoE: CPU with a partial GPU slice)

The expert-offload pattern from upstream llama.cpp carries over: the --override-tensor regex pins the sparse expert (ffn_*_exps) tensors to the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the Q8_0 model each ffn expert tensor is 272 MiB (~4 tensor/block ≈ 1088 MiB/block). With PARALLEL=3 the KV cache tripled, so most expert blocks are pinned to CPU; only blocks 35-40 (output-adjacent, latency- and quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have no exps and are untouched by the regex.

OVERRIDE_EXPERT — the layer-bounded regex listing the blocks whose experts stay on CPU: (0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22| 24|25|26|27|28|29|30|31|32|33|34). Blocks 35-40 (and the always-on full-attention set) are left off the list so their experts load to CUDA.
OVERRIDE_BUFT=CPU.
N_GPU_LAYERS=999 — ik_llama -ngl / --gpu-layers only accepts an integer (the literal all is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the listed exps to CPU.

Fused MoE (fused_moe, fused_up_gate, fused_mmad) is on by default in ik_llama — there is no positive flag; only -no-fmoe disables it. only_active_experts is also on by default. Confirmed in the boot log for the running service.

To move more experts back to the GPU later, drop more block numbers from OVERRIDE_EXPERT (each block is ~1088 MiB on Q8_0; verify nvidia-smi headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV cache there is little headroom, so the default keeps only 35-40 on GPU.

Qwen-Sharp chat template via --jinja

--chat-template-file alone only stores the template text; ik_llama does not render a custom template as real Jinja unless --jinja is also given (inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja fallback>). Without --jinja it warns it only supports "commonly used templates" and silently uses built-in handlers, dropping the Qwen-Sharp template's thinking / image / tool formatting. So the serve script passes both --chat-template-file and --jinja (gated on USE_JINJA, set to 1).

Speculative decoding: disabled (3 parallel slots)

SPEC_TYPE=none (passed as --spec-type none). Speculative decoding is intentionally disabled because ik_llama limits any speculative stage to a a single server slot: with --parallel 3 a configured spec stage hard-exits at load with

ERR [load_model] Speculative decoding is currently limited to a single server slot (-np 1). / n_parallel=3

This service keeps 3 parallel slots (one per user family via PARALLEL=3), so speculative decoding cannot be used here. MTP stays disabled. The model is a low-parameter MoE (3B active of 35B) and does not need the speculation it would cost a slot for.

Lesson. Do not combine speculative decoding with --parallel > 1 in ik_llama. Either drop to -np 1 (one slot) and keep a spec stage, or keep multiple slots and set SPEC_TYPE=none. For a multi-user service the extra slots win.

Flash-attention + sampling

Flash-attention is on by default in ik_llama (flash_attn=true), so no -fa/--flash-attn flag is needed. Sampling, thread, batch, context, and checkpoint tunables (TEMPERATURE, TOP_K, TOP_P, MIN_P, thread, BATCH_SIZE, UBATCH_SIZE, CTX_SIZE, PARALLEL, CTX_CHECKPOINTS, NO_MMAP) are unchanged and pass straight through.

MoE / hybrid-arg reference (not in upstream llama.cpp)

Several ik_llama knobs matter for a CPU-MoE offload host:

flag	meaning	our use
`--split-mode graph`	split tensors+graph across devices; strong for MoE; requires 2+ devices	single 3090 → use `none`
`-ooae` / `--offload-only-active-experts`	default ON: copy only active experts RAM→VRAM	keep ON (huge for A3B: few experts active)
`-grt q8_0`	reduce type for inter-GPU data exchange	not needed (single GPU)
`--fit`	auto-offload to fit VRAM	conflicts with `--override-tensor`; do not use together (runtime error)
`-n-cpu-moe N` / `--cpu-moe`	keep MoE weights of first N / all layers on CPU	also conflicts with `--fit`; our `--override-tensor` is finer-grained
`--fit-margin`	VRAM safety margin for `--fit`	only relevant with `--fit`

The current deployment uses --override-tensor (not --fit or --n-cpu-moe) for the expert offload, so the --fit-conflict rules do not apply.

Rollback

The previous runtime upstream llama.cpp b10121 and its binary (/home/dank/llama.cpp/build/bin/llama-server) are left untouched. To rollback to the old stack, change LLAMA_SERVER= in scripts/frank-zealot.env back to that path and CACHE_TYPE_K/V back to f16 with Hadamard off, and re-enable the ngram spec / --spec-type flags. The model file is not changed either way.

Citations

ikawrakow/ik_llama.cpp README — MoE offload guidance and the --split-mode graph + CPU-offload warning
ik_llama.cpp serverAppConfigs-docs/build.md — build flags incl. GGML_CUDA_FA_ALL_QUANTS and GGML_IQK_FA_ALL_QUANTS
ik_llama.cpp serverAppConfigs-docs/parameters.md — q8_KV (PR 208), K/V Hadamard options, -cuda graphs=0 workaround, and --split-mode graph MoE note (PR 1080)

12 KiB Raw Blame History