--- type: research title: Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18) description: Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a measured prefill cost of nothing (+2.0% on byte-identical traffic). Records the 2026-09-11 amd_iommu=off era that bought a vendor-claimed 13-16% figure never measured here, what that cost, the AMD PerfOpt research (Linux 7.4), the tooling that owns the boot line, and the revert path. tags: - dankclaw - iommu - kernel - grub - prefill - frank-dragoon - halogen - frank-npu timestamp: 2026-09-11 --- # Dankclaw IOMMU translation off — decision record **Current state (since 2026-10-18 01:24): Dankclaw boots `iommu=pt`** — identity DMA, IOMMU armed, XDNA2 NPU bound and usable. Everything below is the record of the 2026-09-11 decision that booted it with **`amd_iommu=off`** on a vendor-claimed 13–16 % prefill figure, what that cost, and the measured flip back: see § *2026-10-18 execution* at the bottom for the numbers and the revert path. ## What changed `GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` on Dankclaw: ``` # before ro quiet splash iommu=pt crashkernel=… # after ro quiet splash amd_iommu=off crashkernel=… ``` Then `sudo update-grub` and a reboot. Nothing else on the command line was touched. ## Evidence The upstream project published the measurement, and it is the only A/B they ran: **460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent**, taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their published serving profile boots this way, so every number in their README is a number with translation off. Decode is unaffected; prefill is where the loss is. The mechanism offered is DMA translation on the path every GPU allocation takes. The project states plainly that this is *their* configuration and not a tuning guide, that the flag turns off DMA translation machine-wide, and that it takes the NPU with it. ## What it costs 1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and `/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure documented in [[frank-npu]]. Audio input left the fleet with it; image input survives on `frank-dragoon` through halogen's vision tower. 2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the observability stack. DMA translation is now off for all of them. This was accepted explicitly as the price of the prefill figure. ## What was deliberately NOT copied The reference machine's command line has six settings. Only one was measured, so only that one was taken: | setting | reference value | why not here | |---|---|---| | `amd_iommu=off` | set | **taken** — the measured 13–16 % | | `amdgpu.gttsize` | `126976` MiB | a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports **114 GiB ≈ 94 % of its 121 GiB** without the flag, so it would buy nothing | | `ttm.pages_limit` | `32505856` | the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists | | `amdgpu.vm_update_mode=0` | set | upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended" | | `amdgpu.noretry=0` | set | as above | | `amdgpu.sg_display=0` | set | as above | ## Verification and revert ```bash cat /proc/cmdline # expect iommu=pt, no amd_iommu=off ls /sys/kernel/iommu_groups/ | wc -l # expect 32 (0 means the flip is gone) cat /sys/kernel/iommu_groups/*/type | sort | uniq -c # 31 identity + 1 DMA (the SD reader) sudo dmesg | grep -m1 amdxdna # driver bound + firmware loaded = NPU alive curl -s http://127.0.0.1:7770/health # frank-dragoon must still answer 200 ``` **To revert:** set `GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off"` in the tracked `dankclaw/host/grub` and run `./scripts/set-dankclaw-bootline.sh --reboot` on Dankclaw. Never hand-edit `/etc/default/grub`: the tracked file is the source, and the installer diffs it, installs it 0644 root:root, runs `update-grub`, and checks the default `grub.cfg` entry before it will reboot. The IOMMU change is independent of the halogen deployment — reverting one does not require reverting the other. ## 2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt) Asked and answered the same day the flip was executed (§ *2026-10-18 execution* below): the figure behind the September decision is on an expiring path, and — the part that actually mattered — it was **never measured on this host**. **Mechanism.** AMD IOMMU **PerfOpt**, spec-defined in the AMD I/O Virtualization Technology (IOMMU) spec §3.4.9: MMIO `0x016C` bit 13 `PERF_OPT_EN`, gated by extended-feature bit 45 `FEATURE_PERF_OPT`. Armed, it stops GPA→SPA translation for the *privileged integrated GPU* — only the IR/IW permission bits are enforced — while the IOMMU stays fully enabled for every other device, the NPU included. Driver: `drivers/iommu/amd/{iommu.c,init.c,amd_iommu_types.h}`. The only consumer is **amdgpu**: `amdgpu_device_use_perfopt()` requires `AMD_IS_APU` **and the identity domain**, and the shared per-IOMMU bit is refcounted so one device's teardown cannot clear it under a peer. **Status.** v2 merged through the iommu tree 2026-09-24, present in `linux-next-20261016`, queued for **Linux 7.4** (merge window late October 2026, stable ~end of year, likely LTS candidate). NOT available here: `/boot/ System.map-7.0.0-38-generic` contains zero `perfopt` symbols, and Ubuntu 26.04's archives cap at `7.0.0-38.38` for both `linux-image-generic` and `linux-generic-hwe-26.04-edge` — reaching 7.4 means a mainline PPA or a local build. A 2026-09-28 follow-up removes the `amdgpu.iommu_perfopt` knob and makes PerfOpt compulsory. **Measured.** Phoronix (2026-09-29, Lemonade + llama.cpp, same kernel, `amdgpu.iommu_perfopt=0` vs default): **18–23 %** on Strix Point / Krackan laptops, **2–4 % on a Strix Halo desktop** — which is also the evidence that our silicon implements the bit (our IOMMU is `1022:1508` Strix/Strix Halo rev 02). **Identity mode is not "no IOMMU".** Ubuntu ships `CONFIG_IOMMU_DEFAULT_DMA_LAZY=y` (STRICT not set), so IOTLB invalidation was already batched, and a coherent iGPU on unified memory takes no SWIOTLB bounce — strict flush and bounce buffering do not explain the September figure. But a device attached to the identity domain still has its DMA walked through 1:1 page tables, and the AMD attach path enables **ATS/PRI/PASID/GCR3** for it; PerfOpt's own enable path has to switch those four off again (the `skip_caps` label in `attach_device`) to get the speedup, and the cover letter says it arms only "when an integrated GPU is **already in identity mode**". So `pt` is most of the recovery and PerfOpt is the last part of it. **The NPU is compatible with identity mode.** `amdxdna` binds through `iommu_sva_bind_device()` + PASID + GCR3 into its own `IOPT_TYPE_SVA` domain, **independent of the default domain type** (`drivers/accel/amdxdna/aie2_pci.c`), and PerfOpt is not on that path at all. It only needs the IOMMU *enabled* — that is the whole `err=-19` story. Kernel pick notes: the Ubuntu 6.17 `-95` SVA failures were a GCR3 regression fixed before our kernel, but **6.18–6.18.7 disables SVA outright**; skip that range. **Container access needs no reboot to settle:** `/dev/accel/accel0` already exists as `crw-rw---- root:render 261,0` and `dank` is in `render`, so a quadlet wants `AddDevice=/dev/accel/accel0` plus `GroupAdd=keep-groups` — the same pattern ComfyUI uses for `/dev/kfd`. **Revised procedure for the next boot-line window** (still a maintenance window, still an explicit OK per run): 1. `iommu=pt` in place of `amd_iommu=off`. The NPU becomes usable at that reboot; the iGPU pays the identity-walk cost until PerfOpt exists on this kernel, which the desktop-class measurement prices in single-digit percent. 2. **Measure `off` vs `pt` on this host, on the current payload** (bench law: stop + mask `frank-dragoon` before any GPU A/B). The 13–16 % was the vendor's A/B against *their* previous boot line, which our record never identified — the 2026-09-11 adoption copied their figure without a local run. The current native-v2 payload does 1,513–1,604 t/s cold prefill, so that baseline is not comparable regardless. 3. Keep the fabric-clock pin (`pp_dpm_fclk` level 7) for GPU+NPU co-run. 4. On 7.4: nothing to do — PerfOpt arms itself when the iGPU is in the identity domain. Confirm with `dmesg | grep -i perfopt`; force off with `amdgpu.iommu_perfopt=0` while the knob still exists. ## 2026-10-18 execution: the flip, measured on this host Executed 01:24–01:35 with Frank's go-ahead. `GRUB_CMDLINE_LINUX_DEFAULT` went from `quiet splash amd_iommu=off` to `quiet splash iommu=pt` through the now-tracked `dankclaw/host/grub` + `scripts/set-dankclaw-bootline.sh` (diff → install 0644 root:root → `update-grub` → verify the default `grub.cfg` entry → reboot only on `--reboot`). **The A/B, on byte-identical traffic.** The corpus is the rebuilt wikitext-2 raw test split — 2,891 lines, **1,287,656 B**, the same file the pre-reboot boot read, and the engine confirms it by billing the same tokens (59,024 prefill prompt, 21,568 needle haystack). Both arms ran `battery-frank-dragoon.py … tail` against the live service with nothing else on the iGPU; the pt arm pre-warmed the 51.2 GB n-gram table with a full read (40 s) so a cold page cache could not masquerade as IOMMU cost. | probe | `amd_iommu=off` | `iommu=pt` | |---|---|---| | cold prefill, 59,024 tokens | 1,585.1 t/s | **1,616.0 t/s (+2.0 %)** | | warm prefill (prompt cache) | 208,531 t/s | 214,131 t/s | | cache ratio | 131.6× | 132.5× | | needle hi/lo @ 21,568 tokens | hit / hit | hit / hit | | decode, 512 sampled tokens | 37.3 t/s | 33.7 t/s | **Prefill cost: none.** Identity mode is not slower here — it is +2.0 %, inside run-to-run spread. The single decode sample moved −9.7 %, and that instrument cannot see anything: three immediate repeats on the same boot gave 31.8, 46.9 and 47.8 t/s on identical requests, so any single-sample decode delta is noise. The 13–16 % that justified `amd_iommu=off` for five weeks does not exist on this host, and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip. **Post-boot gates.** 0 failed units, `systemd` running. 32 IOMMU groups — 31 `identity` + 1 `DMA`, and that one belongs to the Genesys Logic SD card reader (`c2:00.0`, a kernel quirk, not a fleet device). The iGPU (`c5:00.0`) sits in group 20 `identity` with ATS capability; interrupt remapping is enabled. frank-dragoon's boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5 GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all answered 200. (Later the same day the pool was doubled to 401,408 positions for slot parity — that reservation is ~10.8 GiB and the engine-side total ~79.4 GiB; see [[frank-dragoon-flash-server-v2-native]].) **The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/ Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized amdxdna_accel_driver 0.7.0 … on minor 0`, IOMMU group 26 `identity`, PASID cap width 10 (`Exec- Priv-`). `/dev/accel/accel0` is `crw-rw---- root:render 261,0` and `dank` is in `render`, so a quadlet needs `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. **No NPU endpoint is deployed** — [[frank-npu]] stays the history of the retired endpoint; what changed is only its precondition. **Deliberately deferred:** the `pp_dpm_fclk` pin (level 7, 2000 MHz) from the issue's co-run step. Pinning the fabric clock changes iGPU timing for every workload, and nothing co-runs on the NPU yet; it goes in when an NPU endpoint actually exists. PerfOpt still needs Linux 7.4 — `7.0.0-38-generic` has zero `perfopt` symbols and the 26.04 archive caps at `7.0.0-38.38`. **Tooling learned the hard way:** `/tmp` is tmpfs, so the reboot deleted the eval stage mid-A/B. `scripts/stage-frank-dragoon-eval.sh` rebuilds it byte-identically (corpus + probe PNG + battery), and must run through a login shell because `uv` lives in `~/.local/bin`. ## Related - [[frank-npu]] — the endpoint this retired, including the err=-19 root cause - [[halogen-flash-server]] — the runtime bought by the prefill figure - [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target) ## Citations - [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8 - [iommu/amd: Add PerfOpt IOMMU performance optimization support (v2)](https://lists.freedesktop.org/archives/amd-gfx/2026-September/152501.html) — series 0/2 at [/153378.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153378.html), merged via the iommu tree 2026-09-24 ([/153844.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153844.html)) - [Compulsory-PerfOpt follow-up (removes `amdgpu.iommu_perfopt`)](https://patchew.org/linux/20260928045050.955165-1-superm1@kernel.org/) - [Phoronix: AMD Boosting AI/LLM Performance For Radeon iGPUs As Much As 18~23% With Linux 7.4](https://www.phoronix.com/review/amd-perfopt) — Strix Halo desktop arm (2–4 %), `amdgpu.iommu_perfopt=0` A/B; news write-up [here](https://www.phoronix.com/news/AMD-PerfOpt-Linux-7.4) - [Robin Murphy rejecting the AMD passthrough-default patch](https://lkml.org/lkml/2026/3/26/924) — `CONFIG_IOMMU_DEFAULT_PASSTHROUGH` stays the sanctioned path - [Debian wiki: XDNA](https://wiki.debian.org/XDNA) — driver + firmware upstreamed in 6.14; **amdxdna broken on 6.18–6.18.7** by an IOMMU regression that disables SVA ([report](https://lore.kernel.org/regressions/870872aa-28e9-412a-bac6-8020bf560e4f@amd.com/)) - [amd/xdna-driver#704](https://github.com/amd/xdna-driver/issues/704) — the `err=-19` bind failure when the IOMMU is disabled