Frank
301914b1c0
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
101 lines
4.3 KiB
Markdown
101 lines
4.3 KiB
Markdown
---
|
||
type: research
|
||
title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record
|
||
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-large), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
|
||
tags:
|
||
- dankclaw
|
||
- iommu
|
||
- kernel
|
||
- grub
|
||
- prefill
|
||
- frank-large
|
||
- halogen
|
||
- frank-npu
|
||
timestamp: 2026-09-11
|
||
---
|
||
|
||
# Dankclaw IOMMU translation off — decision record
|
||
|
||
**Decision (2026-09-11):** Dankclaw boots with **`amd_iommu=off`** instead of
|
||
`iommu=pt`, to recover the prefill throughput the large-model runtime loses to
|
||
IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is
|
||
retired ([[frank-npu]]).
|
||
|
||
## What changed
|
||
|
||
`GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` on Dankclaw:
|
||
|
||
```
|
||
# before
|
||
ro quiet splash iommu=pt crashkernel=…
|
||
|
||
# after
|
||
ro quiet splash amd_iommu=off crashkernel=…
|
||
```
|
||
|
||
Then `sudo update-grub` and a reboot. Nothing else on the command line was
|
||
touched.
|
||
|
||
## Evidence
|
||
|
||
The upstream project published the measurement, and it is the only A/B they
|
||
ran: **460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent**,
|
||
taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their
|
||
published serving profile boots this way, so every number in their README is a
|
||
number with translation off. Decode is unaffected; prefill is where the loss is.
|
||
|
||
The mechanism offered is DMA translation on the path every GPU allocation
|
||
takes. The project states plainly that this is *their* configuration and not a
|
||
tuning guide, that the flag turns off DMA translation machine-wide, and that it
|
||
takes the NPU with it.
|
||
|
||
## What it costs
|
||
|
||
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
|
||
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
|
||
documented in [[frank-npu]]. Audio input left the fleet with it; image input
|
||
survives on `frank-large` through halogen's vision tower.
|
||
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
|
||
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
|
||
observability stack. DMA translation is now off for all of them. This was
|
||
accepted explicitly as the price of the prefill figure.
|
||
|
||
## What was deliberately NOT copied
|
||
|
||
The reference machine's command line has six settings. Only one was measured,
|
||
so only that one was taken:
|
||
|
||
| setting | reference value | why not here |
|
||
|---|---|---|
|
||
| `amd_iommu=off` | set | **taken** — the measured 13–16 % |
|
||
| `amdgpu.gttsize` | `126976` MiB | a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports **114 GiB ≈ 94 % of its 121 GiB** without the flag, so it would buy nothing |
|
||
| `ttm.pages_limit` | `32505856` | the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists |
|
||
| `amdgpu.vm_update_mode=0` | set | upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended" |
|
||
| `amdgpu.noretry=0` | set | as above |
|
||
| `amdgpu.sg_display=0` | set | as above |
|
||
|
||
## Verification and revert
|
||
|
||
```bash
|
||
cat /proc/cmdline # expect amd_iommu=off, no iommu=pt
|
||
ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30)
|
||
systemctl is-active frank-npu # expect inactive (masked)
|
||
curl -s http://127.0.0.1:7770/health # frank-large must still answer 200
|
||
```
|
||
|
||
**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of
|
||
`amd_iommu=off`, run `sudo update-grub`, reboot, and unmask/start `frank-npu`
|
||
with the archived unit (`.AGENTTRASH/frank-npu.service.flm-npu`) if the NPU
|
||
endpoint is wanted back. The IOMMU change is independent of the halogen
|
||
deployment — reverting one does not require reverting the other.
|
||
|
||
## Related
|
||
|
||
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
|
||
- [[halogen-flash-server]] — the runtime bought by the prefill figure
|
||
- [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target)
|
||
|
||
## Citations
|
||
|
||
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8
|