frank-default-small: run on ik_llama.cpp with q8_KV + Hadamard cache
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121) to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13 toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the old binary is kept for rollback. Runtime changes: - KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard - --split-mode graph (single-GPU no-op, retained per operator pref) - migrate ngram speculation to canonical --spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24 (legacy --spec-ngram-mod-* are rejected in ik main) - N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay pinned to CPU via --override-tensor Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory for the quantized KV cache; recipe + reasoning baked into docs/LLM/ikllama-ornith.md. Tests and both indices updated.
This commit is contained in:
1 parent
c8d7ed65c6
commit
b8de25d70b
6 files changed
+295
-45
No files matched your search
+13
-11
@@ -1,7 +1,7 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
@@ -17,24 +17,24 @@ services run the Qwen-Sharp chat template.
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[upstream llama.cpp b10121
|
||||
B2[ik_llama.cpp build b166e269
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
ngram-mod, selected experts offloaded via override, 491520 ctx]
|
||||
A --> B --> C
|
||||
ngram-mod, q8_KV + Hadamard cache, 491520 ctx]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
I[LLM Clients / API] -->|HTTP :7770| B
|
||||
I -->|HTTP :7770| E
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
@@ -56,15 +56,17 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_KV + Hadamard KV cache,
|
||||
split-mode graph, canonical ngram-mod args, mandatory FA_ALL_QUANTS
|
||||
build flags
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
Reference in new issue
Block a user