docs(iommu): correct the identity-mode cost claim; pt is most of the fix, PerfOpt the last part

Reading patch v2 1/2 verbatim: amd_iommu_enable_perfopt() rehoming needs the
device ALREADY on the identity domain and switches ATS/PRI/PASID/GCR3 off via
the skip_caps path. So an iGPU in identity mode still pays a 1:1 page walk plus
those four capabilities until PerfOpt exists -- iommu=pt recovers most of the
gap, PerfOpt the last part. The first pass claimed pt alone already avoids the
cost; that was wrong and is retracted here and in the posted issue comment
(edit 23).

Also re-keys frank-npu.md, which carried the dead law 'iommu=pt is not a middle
ground' as its retirement rationale: identity mode + SVA is exactly the revival
path (amdxdna binds its own IOPT_TYPE_SVA domain), and the cost is single-digit
percent on this silicon class. Procedure step 2 now demands a local off-vs-pt
measurement instead of inheriting the vendor's 13-16%, whose slow-arm boot line
our record never identified.
This commit is contained in:
Frank
2026-10-18 01:03:19 -07:00
parent 7643381b9b
commit 9b372acde4
4 changed files with 106 additions and 28 deletions
@@ -1,7 +1,7 @@
---
type: research
title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (13-16% prefill for frank-dragoon), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), what was deliberately not copied from the reference machine, and how to revert.
description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (a vendor-claimed 13-16% prefill for frank-dragoon, never measured locally), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), the 2026-10-18 AMD PerfOpt research that puts the trade on an expiring path, and how to revert.
tags:
- dankclaw
- iommu
@@ -92,15 +92,15 @@ deployment — reverting one does not require reverting the other.
## 2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt)
Asked and answered — **the decision changes, the host does not.** Nothing was
changed on Dankclaw today; this section records why the 13–16 % figure is on an
expiring path.
changed on Dankclaw today; this section records why the figure behind the
decision is on an expiring path — and that it was never measured on this host.
**Mechanism.** AMD IOMMU **PerfOpt**, spec-defined in the AMD I/O Virtualization
Technology (IOMMU) spec §3.4.9: MMIO `0x016C` bit 13 `PERF_OPT_EN`, gated by
extended-feature bit 45 `FEATURE_PERF_OPT`. Armed, it stops GPA→SPA translation
for the *privileged integrated GPU* — only the IR/IW permission bits are
enforced — while the IOMMU stays fully enabled for every other device, the NPU
included. Driver: `drivers/iommu/amd/{iommu.c,amd_iommu_types.h}`. The only
included. Driver: `drivers/iommu/amd/{iommu.c,init.c,amd_iommu_types.h}`. The only
consumer is **amdgpu**: `amdgpu_device_use_perfopt()` requires `AMD_IS_APU` **and
the identity domain**, and the shared per-IOMMU bit is refcounted so one
device's teardown cannot clear it under a peer.
@@ -119,13 +119,15 @@ PerfOpt compulsory.
laptops, **2–4 % on a Strix Halo desktop** — which is also the evidence that our
silicon implements the bit (our IOMMU is `1022:1508` Strix/Strix Halo rev 02).
**The September number was never strict-flush cost.** Ubuntu ships
`CONFIG_IOMMU_DEFAULT_DMA_LAZY=y` (STRICT not set), so IOTLB invalidation was
already batched, and a coherent iGPU on unified memory takes no SWIOTLB bounce.
What we measured was **translated-domain** cost — and `iommu=pt` avoids that
today, with or without PerfOpt. The March attempt to make passthrough the AMD
default was rejected outright, so `iommu=pt` / `iommu.passthrough=1` stays the
sanctioned route.
**Identity mode is not "no IOMMU".** Ubuntu ships `CONFIG_IOMMU_DEFAULT_DMA_LAZY=y`
(STRICT not set), so IOTLB invalidation was already batched, and a coherent iGPU
on unified memory takes no SWIOTLB bounce — strict flush and bounce buffering do
not explain the September figure. But a device attached to the identity domain
still has its DMA walked through 1:1 page tables, and the AMD attach path enables
**ATS/PRI/PASID/GCR3** for it; PerfOpt's own enable path has to switch those four
off again (the `skip_caps` label in `attach_device`) to get the speedup, and the
cover letter says it arms only "when an integrated GPU is **already in identity
mode**". So `pt` is most of the recovery and PerfOpt is the last part of it.
**The NPU is compatible with identity mode.** `amdxdna` binds through
`iommu_sva_bind_device()` + PASID + GCR3 into its own `IOPT_TYPE_SVA` domain,
@@ -143,11 +145,15 @@ pattern ComfyUI uses for `/dev/kfd`.
**Revised procedure for the next boot-line window** (still a maintenance window,
still an explicit OK per run):
1. `iommu=pt` in place of `amd_iommu=off` — NPU usable, translated cost gone,
residual 2–4 % on this silicon class.
2. Re-measure prefill against the **current** payload. The 13–16 % belongs to the
2026-09-11 shape; the 2026-10-17 native-v2 boot runs 1,513–1,604 t/s cold
prefill, so the old baseline is not comparable.
1. `iommu=pt` in place of `amd_iommu=off`. The NPU becomes usable at that reboot;
the iGPU pays the identity-walk cost until PerfOpt exists on this kernel, which
the desktop-class measurement prices in single-digit percent.
2. **Measure `off` vs `pt` on this host, on the current payload** (bench law: stop
+ mask `frank-dragoon` before any GPU A/B). The 13–16 % was the vendor's A/B
against *their* previous boot line, which our record never identified — the
2026-09-11 adoption copied their figure without a local run. The current
native-v2 payload does 1,513–1,604 t/s cold prefill, so that baseline is not
comparable regardless.
3. Keep the fabric-clock pin (`pp_dpm_fclk` level 7) for GPU+NPU co-run.
4. On 7.4: nothing to do — PerfOpt arms itself when the iGPU is in the identity
domain. Confirm with `dmesg | grep -i perfopt`; force off with
+15 -3
View File
@@ -42,9 +42,21 @@ upstream project on this exact silicon. That single flag is the whole decision:
out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device").
That is precisely the failure the endpoint was rescued from on 2026-08-30,
recorded in the next section.
- `iommu=pt` is not a middle ground for the NPU *and* the prefill number: the
passthrough domain still costs the prefill throughput the vendor measured.
- The operator chose prefill, so the NPU endpoint goes.
- `iommu=pt` was recorded here as "not a middle ground", on the theory that the
passthrough domain still costs the vendor's prefill number. **That claim did
not survive the 2026-10-18 research.** Identity mode is not free for the iGPU
— its DMA still walks 1:1 page tables and the AMD attach path enables
ATS/PRI/PASID/GCR3 for the device — but on this silicon class the cost is
single-digit percent, and **AMD PerfOpt** (Linux 7.4, merged 2026-09-24)
removes it: the IOMMU then enforces only the IR/IW permission bits for the
privileged iGPU and stays fully armed for every other device, the NPU
included. So `pt` IS the revival path for this endpoint. See
[[dankclaw-iommu-translation-off]] § *2026-10-18 research* for the kernel
census, the versions to avoid (6.18–6.18.7 disable SVA), the device-access
quadlet law (`AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`), and the
measurement step that must precede any decision.
- The operator took the vendor's prefill figure on the September evidence, so the
NPU endpoint went.
Consequences accepted at retirement: **audio input left the fleet with no
replacement** (halogen reads images only, no audio or video), and the surviving