Files

T

Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu

frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.

2026-09-11 14:48:28 -07:00

4.3 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Dankclaw IOMMU translation off — decision record

Decision (2026-09-11): Dankclaw boots with amd_iommu=off instead of iommu=pt, to recover the prefill throughput the large-model runtime loses to IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is retired (frank-npu).

What changed

GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub on Dankclaw:

# before
ro quiet splash iommu=pt crashkernel=…

# after
ro quiet splash amd_iommu=off crashkernel=…

Then sudo update-grub and a reboot. Nothing else on the command line was touched.

Evidence

The upstream project published the measurement, and it is the only A/B they ran: 460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent, taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their published serving profile boots this way, so every number in their README is a number with translation off. Decode is unaffected; prefill is where the loss is.

The mechanism offered is DMA translation on the path every GPU allocation takes. The project states plainly that this is their configuration and not a tuning guide, that the flag turns off DMA translation machine-wide, and that it takes the NPU with it.

What it costs

The NPU endpoint dies. With IOMMU off there is no PASID/SVA state and /dev/accel/accel0 cannot be opened — err=-19, the exact failure documented in frank-npu. Audio input left the fleet with it; image input survives on frank-large through halogen's vision tower.
Posture change machine-wide. Dankclaw also serves Caddy (the TLS edge and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the observability stack. DMA translation is now off for all of them. This was accepted explicitly as the price of the prefill figure.

What was deliberately NOT copied

The reference machine's command line has six settings. Only one was measured, so only that one was taken:

setting	reference value	why not here
`amd_iommu=off`	set	taken — the measured 13–16 %
`amdgpu.gttsize`	`126976` MiB	a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports 114 GiB ≈ 94 % of its 121 GiB without the flag, so it would buy nothing
`ttm.pages_limit`	`32505856`	the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists
`amdgpu.vm_update_mode=0`	set	upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended"
`amdgpu.noretry=0`	set	as above
`amdgpu.sg_display=0`	set	as above

Verification and revert

cat /proc/cmdline                        # expect amd_iommu=off, no iommu=pt
ls /sys/kernel/iommu_groups/ | wc -l     # expect 0 groups (was 30)
systemctl is-active frank-npu            # expect inactive (masked)
curl -s http://127.0.0.1:7770/health     # frank-large must still answer 200

To revert: put iommu=pt back in GRUB_CMDLINE_LINUX_DEFAULT in place of amd_iommu=off, run sudo update-grub, reboot, and unmask/start frank-npu with the archived unit (.AGENTTRASH/frank-npu.service.flm-npu) if the NPU endpoint is wanted back. The IOMMU change is independent of the halogen deployment — reverting one does not require reverting the other.

frank-npu — the endpoint this retired, including the err=-19 root cause
halogen-flash-server — the runtime bought by the prefill figure
kingjones-strix-rocmfp4 — the runtime it replaced (rollback target)

Citations

peonist-ai/halogen-flash-server — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8

4.3 KiB Raw Blame History Unescape Escape