Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template

- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
  native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
  explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
  (replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
  ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
  (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
  retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
  --chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
  update index, deployment, and both system specs; models.json names.

Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
This commit is contained in:
Frank committed 2026-08-24 21:35:11 -07:00
1 parent 189934af15
commit 4462aa992c
14 files changed
+827 -156

No files matched your search

+19 -13
View File
@@ -1,9 +1,10 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on two local machines
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
(frank-default-large).
(2026-08-24): upstream llama.cpp on Dankserver (frank-default-small,
Ornith-1.5-35B-A3B CRACK) and CIRU ROCmFPX llama.cpp on Dankclaw
(frank-default-large, kingjones Qwen3.8-27B ROCmFP4). Both services
run the Qwen-Sharp chat template.
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -18,8 +19,8 @@ graph LR
A[RTX 3090 24GB + 62GB RAM]
B[upstream llama.cpp b10121
frank-default-small, port 7770]
C[Qwen3.6-35B-A3B-Uncensored Q6_K_P
ngram-mod, 38 MoE experts on CPU, 786432 ctx]
C[Ornith-1.5-35B-A3B-CRACK Q8_0
native MTP, MoE experts offloaded via override, 786432 ctx]
A --> B --> C
end
@@ -27,8 +28,8 @@ ngram-mod, 38 MoE experts on CPU, 786432 ctx]
D[gfx1151 iGPU 64 GiB unified memory]
E[CIRU ROCmFPX llama.cpp PromptForge v3
frank-default-large, port 7770]
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
F[Qwen3.8-27B-Uncensored-ROCmFP4 Q6_0 AGENT
MTP draft + Sharp template, 262K ctx / 1 slot]
D --> E --> F
end
@@ -47,8 +48,12 @@ frank-default-large, port 7770]
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
Dankclaw (frank-default-large, active. Restored 2026-08-22)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model
on Dankclaw (frank-default-large, active 2026-08-24)
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-default-small, active 2026-08-24)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
@@ -56,8 +61,9 @@ frank-default-large, port 7770]
## CIRU Reference (Dankclaw)
- [[ciru-activefpx-promptforge]] — the v3 model, ROCm 7.15 toolchain,
and build history (active deployment)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
and build history (on Dankclaw; the slot now serves the kingjones
ROCmFP4 model, see [[kingjones-strix-mtp]])
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
identity, and experiment summary (retired 2026-08-22)
@@ -71,8 +77,8 @@ frank-default-large, port 7770]
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (the model family now served by frank-default-small as
Q6_K_P)
notes and sizing (the historical family; frank-default-small now serves
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
## Memory Management