feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
This commit is contained in:
1 parent
184df9aa0d
commit
301914b1c0
21 files changed
+929
-279
No files matched your search
@@ -1,10 +1,9 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-small,
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX
|
||||
llama.cpp on Dankclaw (frank-large, kingjones777
|
||||
Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template.
|
||||
(2026-09-11): FreeToken v0.1.2 on Dankserver (frank-small,
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and halogen-flash-server 0.5.8 on
|
||||
Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,10 +25,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[kingjones30/ROCmFPX build
|
||||
E[halogen-flash-server 0.5.8
|
||||
frank-large, port 7770]
|
||||
F[kingjones777 Qwen3.8-Flash-Next
|
||||
ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
F[peonist-ai Qwen3.8-Flash-Next
|
||||
w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -53,14 +52,19 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-rocmfp4]] — ACTIVE frank-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11:
|
||||
peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn +
|
||||
quality overlay), rootless Quadlet container on Dankclaw port 7770 —
|
||||
deployed flags, memory model, client contract, troubleshooting
|
||||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||||
the reference machine, and how to revert
|
||||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||||
kingjones30/ROCmFPX fork (swapped 2026-08-30)
|
||||
- [[agentionai-fast-imatrix]] — retired frank-large runbook:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
|
||||
Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38
|
||||
(2026-08-29 → 2026-08-30); rollback binary + payload retained
|
||||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
@@ -78,11 +82,11 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-small)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large,
|
||||
served on the pinned kingjones30/ROCmFPX llama.cpp fork since 2026-08-30)
|
||||
- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on
|
||||
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
|
||||
7771, whole-registry serving, per-request model routing), incl. the
|
||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||
on halogen-flash-server since 2026-09-11)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
every dependent route moved
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
|
||||
Reference in new issue
Block a user