serverAppConfigs/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	Dankclaw IOMMU translation off (amd_iommu=off) — decision record	Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (a vendor-claimed 13-16% prefill for frank-dragoon, never measured locally), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), the 2026-10-18 AMD PerfOpt research that puts the trade on an expiring path, and how to revert.	dankclaw iommu kernel grub prefill frank-dragoon halogen frank-npu	2026-09-11

Dankclaw IOMMU translation off — decision record

Decision (2026-09-11): Dankclaw boots with amd_iommu=off instead of iommu=pt, to recover the prefill throughput the large-model runtime loses to IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is retired (frank-npu).

What changed

GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub on Dankclaw:

# before
ro quiet splash iommu=pt crashkernel=…

# after
ro quiet splash amd_iommu=off crashkernel=…

Then sudo update-grub and a reboot. Nothing else on the command line was touched.

Evidence

The upstream project published the measurement, and it is the only A/B they ran: 460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent, taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their published serving profile boots this way, so every number in their README is a number with translation off. Decode is unaffected; prefill is where the loss is.

The mechanism offered is DMA translation on the path every GPU allocation takes. The project states plainly that this is their configuration and not a tuning guide, that the flag turns off DMA translation machine-wide, and that it takes the NPU with it.

What it costs

1. The NPU endpoint dies. With IOMMU off there is no PASID/SVA state and /dev/accel/accel0 cannot be opened — err=-19, the exact failure documented in frank-npu. Audio input left the fleet with it; image input survives on frank-dragoon through halogen's vision tower.
2. Posture change machine-wide. Dankclaw also serves Caddy (the TLS edge and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the observability stack. DMA translation is now off for all of them. This was accepted explicitly as the price of the prefill figure.

What was deliberately NOT copied

The reference machine's command line has six settings. Only one was measured, so only that one was taken:

setting	reference value	why not here
amd_iommu=off	set	taken — the measured 13–16 %
amdgpu.gttsize	126976 MiB	a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports 114 GiB ≈ 94 % of its 121 GiB without the flag, so it would buy nothing
ttm.pages_limit	32505856	the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists
amdgpu.vm_update_mode=0	set	upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended"
amdgpu.noretry=0	set	as above
amdgpu.sg_display=0	set	as above

Verification and revert

cat /proc/cmdline                        # expect amd_iommu=off, no iommu=pt
ls /sys/kernel/iommu_groups/ | wc -l     # expect 0 groups (was 30)
systemctl is-active frank-npu            # expect inactive (masked)
curl -s http://127.0.0.1:7770/health     # frank-dragoon must still answer 200

To revert: put iommu=pt back in GRUB_CMDLINE_LINUX_DEFAULT in place of amd_iommu=off, run sudo update-grub, reboot, and unmask/start frank-npu with the archived unit (.AGENTTRASH/frank-npu.service.flm-npu) if the NPU endpoint is wanted back. The IOMMU change is independent of the halogen deployment — reverting one does not require reverting the other.

2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt)

Asked and answered — the decision changes, the host does not. Nothing was changed on Dankclaw today; this section records why the figure behind the decision is on an expiring path — and that it was never measured on this host.

Mechanism. AMD IOMMU PerfOpt, spec-defined in the AMD I/O Virtualization Technology (IOMMU) spec §3.4.9: MMIO 0x016C bit 13 PERF_OPT_EN, gated by extended-feature bit 45 FEATURE_PERF_OPT. Armed, it stops GPA→SPA translation for the privileged integrated GPU — only the IR/IW permission bits are enforced — while the IOMMU stays fully enabled for every other device, the NPU included. Driver: drivers/iommu/amd/{iommu.c,init.c,amd_iommu_types.h}. The only consumer is amdgpu: amdgpu_device_use_perfopt() requires AMD_IS_APU and the identity domain, and the shared per-IOMMU bit is refcounted so one device's teardown cannot clear it under a peer.

Status. v2 merged through the iommu tree 2026-09-24, present in linux-next-20261016, queued for Linux 7.4 (merge window late October 2026, stable ~end of year, likely LTS candidate). NOT available here: /boot/ System.map-7.0.0-38-generic contains zero perfopt symbols, and Ubuntu 26.04's archives cap at 7.0.0-38.38 for both linux-image-generic and linux-generic-hwe-26.04-edge — reaching 7.4 means a mainline PPA or a local build. A 2026-09-28 follow-up removes the amdgpu.iommu_perfopt knob and makes PerfOpt compulsory.

Measured. Phoronix (2026-09-29, Lemonade + llama.cpp, same kernel, amdgpu.iommu_perfopt=0 vs default): 18–23 % on Strix Point / Krackan laptops, 2–4 % on a Strix Halo desktop — which is also the evidence that our silicon implements the bit (our IOMMU is 1022:1508 Strix/Strix Halo rev 02).

Identity mode is not "no IOMMU". Ubuntu ships CONFIG_IOMMU_DEFAULT_DMA_LAZY=y (STRICT not set), so IOTLB invalidation was already batched, and a coherent iGPU on unified memory takes no SWIOTLB bounce — strict flush and bounce buffering do not explain the September figure. But a device attached to the identity domain still has its DMA walked through 1:1 page tables, and the AMD attach path enables ATS/PRI/PASID/GCR3 for it; PerfOpt's own enable path has to switch those four off again (the skip_caps label in attach_device) to get the speedup, and the cover letter says it arms only "when an integrated GPU is already in identity mode". So pt is most of the recovery and PerfOpt is the last part of it.

The NPU is compatible with identity mode. amdxdna binds through iommu_sva_bind_device() + PASID + GCR3 into its own IOPT_TYPE_SVA domain, independent of the default domain type (drivers/accel/amdxdna/aie2_pci.c), and PerfOpt is not on that path at all. It only needs the IOMMU enabled — that is the whole err=-19 story. Kernel pick notes: the Ubuntu 6.17 -95 SVA failures were a GCR3 regression fixed before our kernel, but 6.18–6.18.7 disables SVA outright; skip that range.

Container access needs no reboot to settle: /dev/accel/accel0 already exists as crw-rw---- root:render 261,0 and dank is in render, so a quadlet wants AddDevice=/dev/accel/accel0 plus GroupAdd=keep-groups — the same pattern ComfyUI uses for /dev/kfd.

Revised procedure for the next boot-line window (still a maintenance window, still an explicit OK per run):

1. iommu=pt in place of amd_iommu=off. The NPU becomes usable at that reboot; the iGPU pays the identity-walk cost until PerfOpt exists on this kernel, which the desktop-class measurement prices in single-digit percent.
2. Measure off vs pt on this host, on the current payload (bench law: stop
   • mask frank-dragoon before any GPU A/B). The 13–16 % was the vendor's A/B against their previous boot line, which our record never identified — the 2026-09-11 adoption copied their figure without a local run. The current native-v2 payload does 1,513–1,604 t/s cold prefill, so that baseline is not comparable regardless.
3. Keep the fabric-clock pin (pp_dpm_fclk level 7) for GPU+NPU co-run.
4. On 7.4: nothing to do — PerfOpt arms itself when the iGPU is in the identity domain. Confirm with dmesg | grep -i perfopt; force off with amdgpu.iommu_perfopt=0 while the knob still exists.

Related

• frank-npu — the endpoint this retired, including the err=-19 root cause
• halogen-flash-server — the runtime bought by the prefill figure
• kingjones-strix-rocmfp4 — the runtime it replaced (rollback target)

Citations

• peonist-ai/halogen-flash-server — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8
• iommu/amd: Add PerfOpt IOMMU performance optimization support (v2) — series 0/2 at /153378.html, merged via the iommu tree 2026-09-24 (/153844.html)
• Compulsory-PerfOpt follow-up (removes amdgpu.iommu_perfopt)
• Phoronix: AMD Boosting AI/LLM Performance For Radeon iGPUs As Much As 18~23% With Linux 7.4 — Strix Halo desktop arm (2–4 %), amdgpu.iommu_perfopt=0 A/B; news write-up here
• Robin Murphy rejecting the AMD passthrough-default patch — CONFIG_IOMMU_DEFAULT_PASSTHROUGH stays the sanctioned path
• Debian wiki: XDNA — driver + firmware upstreamed in 6.14; amdxdna broken on 6.18–6.18.7 by an IOMMU regression that disables SVA (report)
• amd/xdna-driver#704 — the err=-19 bind failure when the IOMMU is disabled