docs(iommu): AMD PerfOpt removes the iGPU translation penalty (issue #3 research)
Records the 2026-10-18 answer to 'does a recent kernel change retire amd_iommu=off': PerfOpt (iommu tree 2026-09-24, Linux 7.4) stops GPA->SPA translation for the privileged iGPU while the IOMMU stays enabled for every other device, the NPU included. Local census proves the fix is absent here (zero perfopt symbols in System.map-7.0.0-38, archive caps at 7.0.0-38.38), and the September 13-16% figure was translated-domain cost, not strict-flush cost -- lazy DMA is the Ubuntu default, so iommu=pt already avoids it. The corrected procedure therefore flips to iommu=pt now and treats PerfOpt as a 2-4% later bonus on this silicon class (Phoronix desktop arm), with the NPU legal under identity mode because amdxdna binds its own SVA domain. Re-keys the AGENTS.md key-lesson bullet; no host change.
This commit is contained in:
Frank
@@ -1108,9 +1108,15 @@ linked runbooks.
|
||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
||||
ports off the gluetun service.
|
||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-dragoon,
|
||||
2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
|
||||
needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
|
||||
in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
||||
2026-09-11) retires the XDNA2 NPU: the driver's SVA/PASID bind needs an
|
||||
enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). That trade is on an
|
||||
expiring path: **AMD PerfOpt** (iommu tree 2026-09-24, Linux 7.4) stops
|
||||
iGPU GPA→SPA translation while the IOMMU stays enabled for every other
|
||||
device, so the next boot-line window is `iommu=pt` with the NPU usable —
|
||||
measured only 2–4 % residual on a Strix Halo desktop (18–23 % on Strix
|
||||
Point laptops). Not installable here yet: Ubuntu 26.04 caps at
|
||||
`7.0.0-38`. Full research in
|
||||
`serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
||||
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
|
||||
only the CURRENT instance; history across recreations lives in
|
||||
`journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
|
||||
|
||||
@@ -89,6 +89,70 @@ with the archived unit (`.AGENTTRASH/frank-npu.service.flm-npu`) if the NPU
|
||||
endpoint is wanted back. The IOMMU change is independent of the halogen
|
||||
deployment — reverting one does not require reverting the other.
|
||||
|
||||
## 2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt)
|
||||
|
||||
Asked and answered — **the decision changes, the host does not.** Nothing was
|
||||
changed on Dankclaw today; this section records why the 13–16 % figure is on an
|
||||
expiring path.
|
||||
|
||||
**Mechanism.** AMD IOMMU **PerfOpt**, spec-defined in the AMD I/O Virtualization
|
||||
Technology (IOMMU) spec §3.4.9: MMIO `0x016C` bit 13 `PERF_OPT_EN`, gated by
|
||||
extended-feature bit 45 `FEATURE_PERF_OPT`. Armed, it stops GPA→SPA translation
|
||||
for the *privileged integrated GPU* — only the IR/IW permission bits are
|
||||
enforced — while the IOMMU stays fully enabled for every other device, the NPU
|
||||
included. Driver: `drivers/iommu/amd/{iommu.c,amd_iommu_types.h}`. The only
|
||||
consumer is **amdgpu**: `amdgpu_device_use_perfopt()` requires `AMD_IS_APU` **and
|
||||
the identity domain**, and the shared per-IOMMU bit is refcounted so one
|
||||
device's teardown cannot clear it under a peer.
|
||||
|
||||
**Status.** v2 merged through the iommu tree 2026-09-24, present in
|
||||
`linux-next-20261016`, queued for **Linux 7.4** (merge window late October 2026,
|
||||
stable ~end of year, likely LTS candidate). NOT available here: `/boot/
|
||||
System.map-7.0.0-38-generic` contains zero `perfopt` symbols, and Ubuntu 26.04's
|
||||
archives cap at `7.0.0-38.38` for both `linux-image-generic` and
|
||||
`linux-generic-hwe-26.04-edge` — reaching 7.4 means a mainline PPA or a local
|
||||
build. A 2026-09-28 follow-up removes the `amdgpu.iommu_perfopt` knob and makes
|
||||
PerfOpt compulsory.
|
||||
|
||||
**Measured.** Phoronix (2026-09-29, Lemonade + llama.cpp, same kernel,
|
||||
`amdgpu.iommu_perfopt=0` vs default): **18–23 %** on Strix Point / Krackan
|
||||
laptops, **2–4 % on a Strix Halo desktop** — which is also the evidence that our
|
||||
silicon implements the bit (our IOMMU is `1022:1508` Strix/Strix Halo rev 02).
|
||||
|
||||
**The September number was never strict-flush cost.** Ubuntu ships
|
||||
`CONFIG_IOMMU_DEFAULT_DMA_LAZY=y` (STRICT not set), so IOTLB invalidation was
|
||||
already batched, and a coherent iGPU on unified memory takes no SWIOTLB bounce.
|
||||
What we measured was **translated-domain** cost — and `iommu=pt` avoids that
|
||||
today, with or without PerfOpt. The March attempt to make passthrough the AMD
|
||||
default was rejected outright, so `iommu=pt` / `iommu.passthrough=1` stays the
|
||||
sanctioned route.
|
||||
|
||||
**The NPU is compatible with identity mode.** `amdxdna` binds through
|
||||
`iommu_sva_bind_device()` + PASID + GCR3 into its own `IOPT_TYPE_SVA` domain,
|
||||
**independent of the default domain type** (`drivers/accel/amdxdna/aie2_pci.c`),
|
||||
and PerfOpt is not on that path at all. It only needs the IOMMU *enabled* — that
|
||||
is the whole `err=-19` story. Kernel pick notes: the Ubuntu 6.17 `-95` SVA
|
||||
failures were a GCR3 regression fixed before our kernel, but **6.18–6.18.7
|
||||
disables SVA outright**; skip that range.
|
||||
|
||||
**Container access needs no reboot to settle:** `/dev/accel/accel0` already
|
||||
exists as `crw-rw---- root:render 261,0` and `dank` is in `render`, so a quadlet
|
||||
wants `AddDevice=/dev/accel/accel0` plus `GroupAdd=keep-groups` — the same
|
||||
pattern ComfyUI uses for `/dev/kfd`.
|
||||
|
||||
**Revised procedure for the next boot-line window** (still a maintenance window,
|
||||
still an explicit OK per run):
|
||||
|
||||
1. `iommu=pt` in place of `amd_iommu=off` — NPU usable, translated cost gone,
|
||||
residual 2–4 % on this silicon class.
|
||||
2. Re-measure prefill against the **current** payload. The 13–16 % belongs to the
|
||||
2026-09-11 shape; the 2026-10-17 native-v2 boot runs 1,513–1,604 t/s cold
|
||||
prefill, so the old baseline is not comparable.
|
||||
3. Keep the fabric-clock pin (`pp_dpm_fclk` level 7) for GPU+NPU co-run.
|
||||
4. On 7.4: nothing to do — PerfOpt arms itself when the iGPU is in the identity
|
||||
domain. Confirm with `dmesg | grep -i perfopt`; force off with
|
||||
`amdgpu.iommu_perfopt=0` while the knob still exists.
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
|
||||
@@ -98,3 +162,9 @@ deployment — reverting one does not require reverting the other.
|
||||
## Citations
|
||||
|
||||
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8
|
||||
- [iommu/amd: Add PerfOpt IOMMU performance optimization support (v2)](https://lists.freedesktop.org/archives/amd-gfx/2026-September/152501.html) — series 0/2 at [/153378.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153378.html), merged via the iommu tree 2026-09-24 ([/153844.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153844.html))
|
||||
- [Compulsory-PerfOpt follow-up (removes `amdgpu.iommu_perfopt`)](https://patchew.org/linux/20260928045050.955165-1-superm1@kernel.org/)
|
||||
- [Phoronix: AMD Boosting AI/LLM Performance For Radeon iGPUs As Much As 18~23% With Linux 7.4](https://www.phoronix.com/review/amd-perfopt) — Strix Halo desktop arm (2–4 %), `amdgpu.iommu_perfopt=0` A/B; news write-up [here](https://www.phoronix.com/news/AMD-PerfOpt-Linux-7.4)
|
||||
- [Robin Murphy rejecting the AMD passthrough-default patch](https://lkml.org/lkml/2026/3/26/924) — `CONFIG_IOMMU_DEFAULT_PASSTHROUGH` stays the sanctioned path
|
||||
- [Debian wiki: XDNA](https://wiki.debian.org/XDNA) — driver + firmware upstreamed in 6.14; **amdxdna broken on 6.18–6.18.7** by an IOMMU regression that disables SVA ([report](https://lore.kernel.org/regressions/870872aa-28e9-412a-bac6-8020bf560e4f@amd.com/))
|
||||
- [amd/xdna-driver#704](https://github.com/amd/xdna-driver/issues/704) — the `err=-19` bind failure when the IOMMU is disabled
|
||||
|
||||
Reference in New Issue
Block a user