feat(llm): swap frank-large to halogen-flash-server, retire frank-npu

frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
This commit is contained in:
Frank committed 2026-09-11 14:48:28 -07:00
1 parent 184df9aa0d
commit 301914b1c0
21 files changed
+929 -279

No files matched your search

+15 -7
View File
@@ -121,9 +121,17 @@ The LLM inference fleet on the two hosts. Full index:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[kingjones-strix-rocmfp4]] — the ACTIVE frank-large runbook since
2026-08-30: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params,
113.47 GiB) on the card's pinned kingjones30/ROCmFPX build
- [[halogen-flash-server]] — the ACTIVE frank-large runbook since
2026-09-11: peonist-ai/halogen-flash-server 0.5.8 serving
peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
container on Dankclaw, port 7770 — flags, memory model, client contract
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
`amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
(the NPU endpoint), and how to revert
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-large runbook
(2026-08-30 → 2026-09-11) and the rollback target:
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
pinned kingjones30/ROCmFPX build
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
@@ -135,10 +143,10 @@ The LLM inference fleet on the two hosts. Full index:
- [[Dankserver System Specs]] — the RTX 3090 box hardware
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
- [[frank-npu]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw:
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; whole local
registry served on one port with per-request model routing; includes the
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
retirement record, including the IOMMU/SVA err=-19 root cause and
where every dependent route moved
### Benchmarks and Sizing