Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns.
This commit is contained in:
1 parent
189934af15
commit
4462aa992c
14 files changed
+827
-156
No files matched your search
+19
-13
@@ -1,9 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-22): upstream llama.cpp on Dankserver (frank-default-small) and
|
||||
CIRU ROCmFPX llama.cpp with the PromptForge v3 runtime on Dankclaw
|
||||
(frank-default-large).
|
||||
(2026-08-24): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and CIRU ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8-27B ROCmFP4). Both services
|
||||
run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -18,8 +19,8 @@ graph LR
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[upstream llama.cpp b10121
|
||||
frank-default-small, port 7770]
|
||||
C[Qwen3.6-35B-A3B-Uncensored Q6_K_P
|
||||
ngram-mod, 38 MoE experts on CPU, 786432 ctx]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q8_0
|
||||
native MTP, MoE experts offloaded via override, 786432 ctx]
|
||||
A --> B --> C
|
||||
end
|
||||
|
||||
@@ -27,8 +28,8 @@ ngram-mod, 38 MoE experts on CPU, 786432 ctx]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[CIRU ROCmFPX llama.cpp PromptForge v3
|
||||
frank-default-large, port 7770]
|
||||
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3
|
||||
262K ctx / 1 slot, full-W8 views, Output-K8 MTP, BF16 vision]
|
||||
F[Qwen3.8-27B-Uncensored-ROCmFP4 Q6_0 AGENT
|
||||
MTP draft + Sharp template, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
@@ -47,8 +48,12 @@ frank-default-large, port 7770]
|
||||
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[ciru-activefpx-promptforge]] — the v3 CIRU PromptForge runtime on
|
||||
Dankclaw (frank-default-large, active. Restored 2026-08-22)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model
|
||||
on Dankclaw (frank-default-large, active 2026-08-24)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the upstream llama.cpp box (frank-default-small)
|
||||
@@ -56,8 +61,9 @@ frank-default-large, port 7770]
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the v3 model, ROCm 7.15 toolchain,
|
||||
and build history (active deployment)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the slot now serves the kingjones
|
||||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
@@ -71,8 +77,8 @@ frank-default-large, port 7770]
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (the model family now served by frank-default-small as
|
||||
Q6_K_P)
|
||||
notes and sizing (the historical family; frank-default-small now serves
|
||||
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
|
||||
|
||||
## Memory Management
|
||||
|
||||
|
||||
Reference in new issue
Block a user