dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2a124876538f9cc612378a7ad4c046b0f521dad0
serverAppConfigs/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

244 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: reference
title: Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18)
description: "Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line."
tags:
- dankclaw
- iommu
- kernel
- grub
- prefill
- frank-dragoon
- halogen
- frank-npu
timestamp: 2026-09-11
---
# Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18)
**Current state (since 2026-10-18 01:24): Dankclaw boots `iommu=pt`** — identity
DMA, IOMMU armed, XDNA2 NPU bound and usable. Everything below is the record of
the 2026-09-11 decision that booted it with **`amd_iommu=off`** on a vendor-claimed
13–16 % prefill figure, what that cost, and the measured flip back: see
§ *2026-10-18 execution* at the bottom for the numbers and the revert path.
## What changed
`GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` on Dankclaw:
```
# before
ro quiet splash iommu=pt crashkernel=…
# after
ro quiet splash amd_iommu=off crashkernel=…
```
Then `sudo update-grub` and a reboot. Nothing else on the command line was
touched.
## Evidence
The upstream project published the measurement, and it is the only A/B they
ran: **460 → 385 tok/s prefill at a 2,048-token prompt, 13 to 16 percent**,
taken on a Ryzen AI Max+ 395 with 128 GB — the same SKU as Dankclaw. Their
published serving profile boots this way, so every number in their README is a
number with translation off. Decode is unaffected; prefill is where the loss is.
The mechanism offered is DMA translation on the path every GPU allocation
takes. The project states plainly that this is *their* configuration and not a
tuning guide, that the flag turns off DMA translation machine-wide, and that it
takes the NPU with it.
## What it costs
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
documented in [[frank-npu]]. Audio input left the fleet with it; image input
survives on `frank-dragoon` through halogen's vision tower.
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
observability stack. DMA translation is now off for all of them. This was
accepted explicitly as the price of the prefill figure.
## What was deliberately NOT copied
The reference machine's command line has six settings. Only one was measured,
so only that one was taken:
| setting | reference value | why not here |
|---|---|---|
| `amd_iommu=off` | set | **taken** — the measured 13–16 % |
| `amdgpu.gttsize` | `126976` MiB | a size, not a constant: it must equal installed RAM (their table: 128 GB → 126976, 96 GB → 95232, 64 GB → 63488). Dankclaw's GTT already reports **114 GiB ≈ 94 % of its 121 GiB** without the flag, so it would buy nothing |
| `ttm.pages_limit` | `32505856` | the same number in 4 KiB pages (their 128 GB row is 99.3 % of RAM). Pasting it onto a different machine asks the driver for more GTT than exists |
| `amdgpu.vm_update_mode=0` | set | upstream has never run without it, makes no claim about what it buys, and lists it "for completeness rather than recommended" |
| `amdgpu.noretry=0` | set | as above |
| `amdgpu.sg_display=0` | set | as above |
## Verification and revert
```bash
cat /proc/cmdline # expect iommu=pt, no amd_iommu=off
ls /sys/kernel/iommu_groups/ | wc -l # expect 32 (0 means the flip is gone)
cat /sys/kernel/iommu_groups/*/type | sort | uniq -c # 31 identity + 1 DMA (the SD reader)
sudo dmesg | grep -m1 amdxdna # driver bound + firmware loaded = NPU alive
curl -s http://127.0.0.1:7770/health # frank-dragoon must still answer 200
```
**To revert:** set `GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off"` in
the tracked `dankclaw/host/grub` and run `./scripts/set-dankclaw-bootline.sh
--reboot` on Dankclaw. Never hand-edit `/etc/default/grub`: the tracked file is
the source, and the installer diffs it, installs it 0644 root:root, runs
`update-grub`, and checks the default `grub.cfg` entry before it will reboot. The
IOMMU change is independent of the halogen deployment — reverting one does not
require reverting the other.
## 2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt)
Asked and answered the same day the flip was executed (§ *2026-10-18 execution*
below): the figure behind the September decision is on an expiring path, and — the
part that actually mattered — it was **never measured on this host**.
**Mechanism.** AMD IOMMU **PerfOpt**, spec-defined in the AMD I/O Virtualization
Technology (IOMMU) spec §3.4.9: MMIO `0x016C` bit 13 `PERF_OPT_EN`, gated by
extended-feature bit 45 `FEATURE_PERF_OPT`. Armed, it stops GPA→SPA translation
for the *privileged integrated GPU* — only the IR/IW permission bits are
enforced — while the IOMMU stays fully enabled for every other device, the NPU
included. Driver: `drivers/iommu/amd/{iommu.c,init.c,amd_iommu_types.h}`. The only
consumer is **amdgpu**: `amdgpu_device_use_perfopt()` requires `AMD_IS_APU` **and
the identity domain**, and the shared per-IOMMU bit is refcounted so one
device's teardown cannot clear it under a peer.
**Status.** v2 merged through the iommu tree 2026-09-24, present in
`linux-next-20261016`, queued for **Linux 7.4** (merge window late October 2026,
stable ~end of year, likely LTS candidate). NOT available here: `/boot/
System.map-7.0.0-38-generic` contains zero `perfopt` symbols, and Ubuntu 26.04's
archives cap at `7.0.0-38.38` for both `linux-image-generic` and
`linux-generic-hwe-26.04-edge` — reaching 7.4 means a mainline PPA or a local
build. A 2026-09-28 follow-up removes the `amdgpu.iommu_perfopt` knob and makes
PerfOpt compulsory.
**Measured.** Phoronix (2026-09-29, Lemonade + llama.cpp, same kernel,
`amdgpu.iommu_perfopt=0` vs default): **18–23 %** on Strix Point / Krackan
laptops, **2–4 % on a Strix Halo desktop** — which is also the evidence that our
silicon implements the bit (our IOMMU is `1022:1508` Strix/Strix Halo rev 02).
**Identity mode is not "no IOMMU".** Ubuntu ships `CONFIG_IOMMU_DEFAULT_DMA_LAZY=y`
(STRICT not set), so IOTLB invalidation was already batched, and a coherent iGPU
on unified memory takes no SWIOTLB bounce — strict flush and bounce buffering do
not explain the September figure. But a device attached to the identity domain
still has its DMA walked through 1:1 page tables, and the AMD attach path enables
**ATS/PRI/PASID/GCR3** for it; PerfOpt's own enable path has to switch those four
off again (the `skip_caps` label in `attach_device`) to get the speedup, and the
cover letter says it arms only "when an integrated GPU is **already in identity
mode**". So `pt` is most of the recovery and PerfOpt is the last part of it.
**The NPU is compatible with identity mode.** `amdxdna` binds through
`iommu_sva_bind_device()` + PASID + GCR3 into its own `IOPT_TYPE_SVA` domain,
**independent of the default domain type** (`drivers/accel/amdxdna/aie2_pci.c`),
and PerfOpt is not on that path at all. It only needs the IOMMU *enabled* — that
is the whole `err=-19` story. Kernel pick notes: the Ubuntu 6.17 `-95` SVA
failures were a GCR3 regression fixed before our kernel, but **6.18–6.18.7
disables SVA outright**; skip that range.
**Container access needs no reboot to settle:** `/dev/accel/accel0` already
exists as `crw-rw---- root:render 261,0` and `dank` is in `render`, so a quadlet
wants `AddDevice=/dev/accel/accel0` plus `GroupAdd=keep-groups` — the same
pattern ComfyUI uses for `/dev/kfd`.
**Revised procedure for the next boot-line window** (still a maintenance window,
still an explicit OK per run):
1. `iommu=pt` in place of `amd_iommu=off`. The NPU becomes usable at that reboot;
the iGPU pays the identity-walk cost until PerfOpt exists on this kernel, which
the desktop-class measurement prices in single-digit percent.
2. **Measure `off` vs `pt` on this host, on the current payload** (bench law: stop
+ mask `frank-dragoon` before any GPU A/B). The 13–16 % was the vendor's A/B
against *their* previous boot line, which our record never identified — the
2026-09-11 adoption copied their figure without a local run. The current
native-v2 payload does 1,513–1,604 t/s cold prefill, so that baseline is not
comparable regardless.
3. Keep the fabric-clock pin (`pp_dpm_fclk` level 7) for GPU+NPU co-run.
4. On 7.4: nothing to do — PerfOpt arms itself when the iGPU is in the identity
domain. Confirm with `dmesg | grep -i perfopt`; force off with
`amdgpu.iommu_perfopt=0` while the knob still exists.
## 2026-10-18 execution: the flip, measured on this host
Executed 01:24–01:35 with Frank's go-ahead. `GRUB_CMDLINE_LINUX_DEFAULT` went from
`quiet splash amd_iommu=off` to `quiet splash iommu=pt` through the now-tracked
`dankclaw/host/grub` + `scripts/set-dankclaw-bootline.sh` (diff → install 0644
root:root → `update-grub` → verify the default `grub.cfg` entry → reboot only on
`--reboot`).
**The A/B, on byte-identical traffic.** The corpus is the rebuilt wikitext-2 raw
test split — 2,891 lines, **1,287,656 B**, the same file the pre-reboot boot read,
and the engine confirms it by billing the same tokens (59,024 prefill prompt,
21,568 needle haystack). Both arms ran `battery-frank-dragoon.py … tail` against
the live service with nothing else on the iGPU; the pt arm pre-warmed the 51.2 GB
n-gram table with a full read (40 s) so a cold page cache could not masquerade as
IOMMU cost.
| probe | `amd_iommu=off` | `iommu=pt` |
|---|---|---|
| cold prefill, 59,024 tokens | 1,585.1 t/s | **1,616.0 t/s (+2.0 %)** |
| warm prefill (prompt cache) | 208,531 t/s | 214,131 t/s |
| cache ratio | 131.6× | 132.5× |
| needle hi/lo @ 21,568 tokens | hit / hit | hit / hit |
| decode, 512 sampled tokens | 37.3 t/s | 33.7 t/s |
**Prefill cost: none.** Identity mode is not slower here — it is +2.0 %, inside
run-to-run spread. The single decode sample moved −9.7 %, and that instrument
cannot see anything: three immediate repeats on the same boot gave 31.8, 46.9 and
47.8 t/s on identical requests, so any single-sample decode delta is noise. The
13–16 % that justified `amd_iommu=off` for five weeks does not exist on this host,
and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip.
**Post-boot gates.** 0 failed units, `systemd` running. 32 IOMMU groups — 31
`identity` + 1 `DMA`, and that one belongs to the Genesys Logic SD card reader
(`c2:00.0`, a kernel quirk, not a fleet device). The iGPU (`c5:00.0`) sits in group
20 `identity` with ATS capability; interrupt remapping is enabled. frank-dragoon's
boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working),
see [[frank-dragoon]].)
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized
amdxdna_accel_driver 0.7.0 … on minor 0`, IOMMU group 26 `identity`, PASID cap width
10 (`Exec- Priv-`). `/dev/accel/accel0` is `crw-rw---- root:render 261,0` and `dank`
is in `render`, so a quadlet needs `AddDevice=/dev/accel/accel0` +
`GroupAdd=keep-groups`. **No NPU endpoint is deployed** — [[frank-npu]] stays the
history of the retired endpoint; what changed is only its precondition.
**Deliberately deferred:** the `pp_dpm_fclk` pin (level 7, 2000 MHz) from the issue's
co-run step. Pinning the fabric clock changes iGPU timing for every workload, and
nothing co-runs on the NPU yet; it goes in when an NPU endpoint actually exists.
PerfOpt still needs Linux 7.4 — `7.0.0-38-generic` has zero `perfopt` symbols and the
26.04 archive caps at `7.0.0-38.38`.
**Tooling learned the hard way:** `/tmp` is tmpfs, so the reboot deleted the eval
stage mid-A/B. `scripts/stage-frank-dragoon-eval.sh` rebuilds it byte-identically
(corpus + probe PNG + battery), and must run through a login shell because `uv`
lives in `~/.local/bin`.
## Related
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
- [[frank-dragoon]] — the runtime bought by the prefill figure
- [[frank-dragoon]] — the runtime it replaced (rollback target)
## Citations
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README "The host settings these numbers were measured on", CHANGELOG 0.5.5–0.5.8
- [iommu/amd: Add PerfOpt IOMMU performance optimization support (v2)](https://lists.freedesktop.org/archives/amd-gfx/2026-September/152501.html) — series 0/2 at [/153378.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153378.html), merged via the iommu tree 2026-09-24 ([/153844.html](https://lists.freedesktop.org/archives/amd-gfx/2026-September/153844.html))
- [Compulsory-PerfOpt follow-up (removes `amdgpu.iommu_perfopt`)](https://patchew.org/linux/20260928045050.955165-1-superm1@kernel.org/)
- [Phoronix: AMD Boosting AI/LLM Performance For Radeon iGPUs As Much As 18~23% With Linux 7.4](https://www.phoronix.com/review/amd-perfopt) — Strix Halo desktop arm (2–4 %), `amdgpu.iommu_perfopt=0` A/B; news write-up [here](https://www.phoronix.com/news/AMD-PerfOpt-Linux-7.4)
- [Robin Murphy rejecting the AMD passthrough-default patch](https://lkml.org/lkml/2026/3/26/924) — `CONFIG_IOMMU_DEFAULT_PASSTHROUGH` stays the sanctioned path
- [Debian wiki: XDNA](https://wiki.debian.org/XDNA) — driver + firmware upstreamed in 6.14; **amdxdna broken on 6.18–6.18.7** by an IOMMU regression that disables SVA ([report](https://lore.kernel.org/regressions/870872aa-28e9-412a-bac6-8020bf560e4f@amd.com/))
- [amd/xdna-driver#704](https://github.com/amd/xdna-driver/issues/704) — the `err=-19` bind failure when the IOMMU is disabled
Reference in new issue View Git Blame Copy Permalink