frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache

Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.

Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
  --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
  (legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
  pinned to CPU via --override-tensor

Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
This commit is contained in:
Frank committed 2026-08-26 02:59:34 -07:00
1 parent c8d7ed65c6
commit b8de25d70b
6 files changed
+295 -45

No files matched your search

+13 -11
View File
@@ -1,7 +1,7 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on two local machines
(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
services run the Qwen-Sharp chat template.
@@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template.
graph LR
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
A[RTX 3090 24GB + 62GB RAM]
B[upstream llama.cpp b10121
B2[ik_llama.cpp build b166e269
frank-default-small, port 7770]
C[Ornith-1.5-35B-A3B-CRACK Q6_K
ngram-mod, selected experts offloaded via override, 491520 ctx]
A --> B --> C
ngram-mod, q8_KV + Hadamard cache, 491520 ctx]
A --> B2 --> C
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 iGPU 64 GiB unified memory]
E2[gfx1151 iGPU 64 GiB unified memory]
E[charlie12345/ROCmFPX build-kairic (v81)
frank-default-large, port 7770]
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
D --> E --> F
E2 --> E --> F
end
I[LLM Clients / API] -->|HTTP :7770| B
I -->|HTTP :7770| E
CLI[LLM Clients / API] -->|HTTP :7770| B2
CLI -->|HTTP :7770| E
```
## Servers
@@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
on Dankclaw (retired)
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-default-small, active 2026-08-24)
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
frank-default-small (2026-08-26): q8_KV + Hadamard KV cache,
split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS
build flags
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-default-small, active 2026-08-24)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
## CIRU Reference (Dankclaw)