diff --git a/AGENTS.md b/AGENTS.md index 3a6bb07..734816c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1107,17 +1107,17 @@ linked runbooks. times (`RuntimeError: No HIP GPUs are available`). - SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service. -- `amd_iommu=off` on Dankclaw (adopted 2026-09-11 on the vendor's 13–16 % - prefill figure, never measured here) retires the XDNA2 NPU: the driver's - SVA/PASID bind needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). - That trade is on an expiring path: **AMD PerfOpt** (iommu tree 2026-09-24, - Linux 7.4) makes the IOMMU skip GPA→SPA translation for the *privileged iGPU* - while staying fully armed for every other device — and it arms only when the - iGPU is already in the identity domain, so the next boot-line window is - `iommu=pt` with the NPU usable, not `off`. PerfOpt buys 2–4 % back on a Strix - Halo desktop (18–23 % on Strix Point laptops); it is not installable here - yet (Ubuntu 26.04 caps at `7.0.0-38`). Research in - `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`. +- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in + `dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh` + (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by + hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11), + adopted on the vendor's 13–16 % prefill figure and never measured here: the + local A/B on byte-identical traffic says identity mode costs **nothing** + (+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free — + `amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old + `err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree + 2026-09-24, Linux 7.4) still skips the page walk outright for the privileged + iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`. - Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers only the CURRENT instance; history across recreations lives in `journalctl --user -u .service`. Read rootless storage WITHOUT sudo diff --git a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md index d71cadc..f635813 100644 --- a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md +++ b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md @@ -1,7 +1,7 @@ --- type: research -title: Dankclaw IOMMU translation off (amd_iommu=off) — decision record -description: Why Dankclaw boots with amd_iommu=off since 2026-09-11, what it buys (a vendor-claimed 13-16% prefill for frank-dragoon, never measured locally), what it costs (the XDNA2 NPU endpoint, machine-wide DMA translation), the 2026-10-18 AMD PerfOpt research that puts the trade on an expiring path, and how to revert. +title: Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18) +description: Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a measured prefill cost of nothing (+2.0% on byte-identical traffic). Records the 2026-09-11 amd_iommu=off era that bought a vendor-claimed 13-16% figure never measured here, what that cost, the AMD PerfOpt research (Linux 7.4), the tooling that owns the boot line, and the revert path. tags: - dankclaw - iommu @@ -16,10 +16,11 @@ timestamp: 2026-09-11 # Dankclaw IOMMU translation off — decision record -**Decision (2026-09-11):** Dankclaw boots with **`amd_iommu=off`** instead of -`iommu=pt`, to recover the prefill throughput the large-model runtime loses to -IOMMU translation. The XDNA2 NPU endpoint was accepted as the cost and is -retired ([[frank-npu]]). +**Current state (since 2026-10-18 01:24): Dankclaw boots `iommu=pt`** — identity +DMA, IOMMU armed, XDNA2 NPU bound and usable. Everything below is the record of +the 2026-09-11 decision that booted it with **`amd_iommu=off`** on a vendor-claimed +13–16 % prefill figure, what that cost, and the measured flip back: see +§ *2026-10-18 execution* at the bottom for the numbers and the revert path. ## What changed @@ -77,23 +78,26 @@ so only that one was taken: ## Verification and revert ```bash -cat /proc/cmdline # expect amd_iommu=off, no iommu=pt -ls /sys/kernel/iommu_groups/ | wc -l # expect 0 groups (was 30) -systemctl is-active frank-npu # expect inactive (masked) +cat /proc/cmdline # expect iommu=pt, no amd_iommu=off +ls /sys/kernel/iommu_groups/ | wc -l # expect 32 (0 means the flip is gone) +cat /sys/kernel/iommu_groups/*/type | sort | uniq -c # 31 identity + 1 DMA (the SD reader) +sudo dmesg | grep -m1 amdxdna # driver bound + firmware loaded = NPU alive curl -s http://127.0.0.1:7770/health # frank-dragoon must still answer 200 ``` -**To revert:** put `iommu=pt` back in `GRUB_CMDLINE_LINUX_DEFAULT` in place of -`amd_iommu=off`, run `sudo update-grub`, reboot, and unmask/start `frank-npu` -with the archived unit (`.AGENTTRASH/frank-npu.service.flm-npu`) if the NPU -endpoint is wanted back. The IOMMU change is independent of the halogen -deployment — reverting one does not require reverting the other. +**To revert:** set `GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off"` in +the tracked `dankclaw/host/grub` and run `./scripts/set-dankclaw-bootline.sh +--reboot` on Dankclaw. Never hand-edit `/etc/default/grub`: the tracked file is +the source, and the installer diffs it, installs it 0644 root:root, runs +`update-grub`, and checks the default `grub.cfg` entry before it will reboot. The +IOMMU change is independent of the halogen deployment — reverting one does not +require reverting the other. ## 2026-10-18 research: the penalty now has a hardware fix (AMD PerfOpt) -Asked and answered — **the decision changes, the host does not.** Nothing was -changed on Dankclaw today; this section records why the figure behind the -decision is on an expiring path — and that it was never measured on this host. +Asked and answered the same day the flip was executed (§ *2026-10-18 execution* +below): the figure behind the September decision is on an expiring path, and — the +part that actually mattered — it was **never measured on this host**. **Mechanism.** AMD IOMMU **PerfOpt**, spec-defined in the AMD I/O Virtualization Technology (IOMMU) spec §3.4.9: MMIO `0x016C` bit 13 `PERF_OPT_EN`, gated by @@ -159,6 +163,65 @@ still an explicit OK per run): domain. Confirm with `dmesg | grep -i perfopt`; force off with `amdgpu.iommu_perfopt=0` while the knob still exists. +## 2026-10-18 execution: the flip, measured on this host + +Executed 01:24–01:35 with Frank's go-ahead. `GRUB_CMDLINE_LINUX_DEFAULT` went from +`quiet splash amd_iommu=off` to `quiet splash iommu=pt` through the now-tracked +`dankclaw/host/grub` + `scripts/set-dankclaw-bootline.sh` (diff → install 0644 +root:root → `update-grub` → verify the default `grub.cfg` entry → reboot only on +`--reboot`). + +**The A/B, on byte-identical traffic.** The corpus is the rebuilt wikitext-2 raw +test split — 2,891 lines, **1,287,656 B**, the same file the pre-reboot boot read, +and the engine confirms it by billing the same tokens (59,024 prefill prompt, +21,568 needle haystack). Both arms ran `battery-frank-dragoon.py … tail` against +the live service with nothing else on the iGPU; the pt arm pre-warmed the 51.2 GB +n-gram table with a full read (40 s) so a cold page cache could not masquerade as +IOMMU cost. + +| probe | `amd_iommu=off` | `iommu=pt` | +|---|---|---| +| cold prefill, 59,024 tokens | 1,585.1 t/s | **1,616.0 t/s (+2.0 %)** | +| warm prefill (prompt cache) | 208,531 t/s | 214,131 t/s | +| cache ratio | 131.6× | 132.5× | +| needle hi/lo @ 21,568 tokens | hit / hit | hit / hit | +| decode, 512 sampled tokens | 37.3 t/s | 33.7 t/s | + +**Prefill cost: none.** Identity mode is not slower here — it is +2.0 %, inside +run-to-run spread. The single decode sample moved −9.7 %, and that instrument +cannot see anything: three immediate repeats on the same boot gave 31.8, 46.9 and +47.8 t/s on identical requests, so any single-sample decode delta is noise. The +13–16 % that justified `amd_iommu=off` for five weeks does not exist on this host, +and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip. + +**Post-boot gates.** 0 failed units, `systemd` running. 32 IOMMU groups — 31 +`identity` + 1 `DMA`, and that one belongs to the Genesys Logic SD card reader +(`c2:00.0`, a kernel quirk, not a fleet device). The iGPU (`c5:00.0`) sits in group +20 `identity` with ATS capability; interrupt remapping is enabled. frank-dragoon's +boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5 +GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP +depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all +answered 200. + +**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/ +Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized +amdxdna_accel_driver 0.7.0 … on minor 0`, IOMMU group 26 `identity`, PASID cap width +10 (`Exec- Priv-`). `/dev/accel/accel0` is `crw-rw---- root:render 261,0` and `dank` +is in `render`, so a quadlet needs `AddDevice=/dev/accel/accel0` + +`GroupAdd=keep-groups`. **No NPU endpoint is deployed** — [[frank-npu]] stays the +history of the retired endpoint; what changed is only its precondition. + +**Deliberately deferred:** the `pp_dpm_fclk` pin (level 7, 2000 MHz) from the issue's +co-run step. Pinning the fabric clock changes iGPU timing for every workload, and +nothing co-runs on the NPU yet; it goes in when an NPU endpoint actually exists. +PerfOpt still needs Linux 7.4 — `7.0.0-38-generic` has zero `perfopt` symbols and the +26.04 archive caps at `7.0.0-38.38`. + +**Tooling learned the hard way:** `/tmp` is tmpfs, so the reboot deleted the eval +stage mid-A/B. `scripts/stage-frank-dragoon-eval.sh` rebuilds it byte-identically +(corpus + probe PNG + battery), and must run through a login shell because `uv` +lives in `~/.local/bin`. + ## Related - [[frank-npu]] — the endpoint this retired, including the err=-19 root cause diff --git a/serverAppConfigs-docs/LLM/frank-npu.md b/serverAppConfigs-docs/LLM/frank-npu.md index fc86322..b938480 100644 --- a/serverAppConfigs-docs/LLM/frank-npu.md +++ b/serverAppConfigs-docs/LLM/frank-npu.md @@ -43,20 +43,25 @@ upstream project on this exact silicon. That single flag is the whole decision: That is precisely the failure the endpoint was rescued from on 2026-08-30, recorded in the next section. - `iommu=pt` was recorded here as "not a middle ground", on the theory that the - passthrough domain still costs the vendor's prefill number. **That claim did - not survive the 2026-10-18 research.** Identity mode is not free for the iGPU - — its DMA still walks 1:1 page tables and the AMD attach path enables - ATS/PRI/PASID/GCR3 for the device — but on this silicon class the cost is - single-digit percent, and **AMD PerfOpt** (Linux 7.4, merged 2026-09-24) - removes it: the IOMMU then enforces only the IR/IW permission bits for the - privileged iGPU and stays fully armed for every other device, the NPU - included. So `pt` IS the revival path for this endpoint. See - [[dankclaw-iommu-translation-off]] § *2026-10-18 research* for the kernel - census, the versions to avoid (6.18–6.18.7 disable SVA), the device-access - quadlet law (`AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`), and the - measurement step that must precede any decision. + passthrough domain still costs the vendor's prefill number. **That claim is + dead: the host flipped to `iommu=pt` on 2026-10-18 and the local A/B on + byte-identical traffic measured the cost at nothing** (+2.0 % cold prefill, + 1,616 t/s at 59k tokens). Identity mode does keep a 1:1 page walk plus + ATS/PRI/PASID/GCR3 armed for the iGPU — worth 2–4 % on this silicon class, and + **AMD PerfOpt** (Linux 7.4, merged 2026-09-24) removes it outright by enforcing + only the IR/IW permission bits for the privileged iGPU — but nothing waits on + that now. +- **The precondition is satisfied; the endpoint is still not deployed.** `amdxdna` + binds `0000:c6:00.1` (driver 0.7.0, firmware `amdnpu/17f0_11/npu_7.sbin`), the + NPU sits in IOMMU group 26 `identity` with PASID width 10, `/dev/accel/accel0` + is `crw-rw---- root:render 261,0` and `dank` is in `render` — so a revival + quadlet needs `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`, the + ComfyUI `/dev/kfd` pattern. Kernel versions to avoid: **6.18–6.18.7** disables + SVA outright. Full census in [[dankclaw-iommu-translation-off]] + § *2026-10-18 execution*. - The operator took the vendor's prefill figure on the September evidence, so the - NPU endpoint went. + NPU endpoint went — and it stays retired because nobody has asked for a model + on it, not because the hardware is unreachable. Consequences accepted at retirement: **audio input left the fleet with no replacement** (halogen reads images only, no audio or video), and the surviving diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index b9a4ae9..f55dbc4 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -463,10 +463,12 @@ frank-zealot, port 7771] mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and decode root-cause chain are the record of why the fleet went back to halogen -- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with - `amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream), - what it cost (the NPU endpoint), what was deliberately not copied from - the reference machine, and how to revert +- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: **`iommu=pt`** + since 2026-10-18, measured on this host at no prefill cost (+2.0 %, 1,616 t/s + cold), which restored the XDNA2 NPU; the `amd_iommu=off` era (2026-09-11) it + replaced ran on a 13-16 % figure measured upstream only; includes the AMD + PerfOpt research, what was deliberately not copied from the reference machine, + and the revert path via `scripts/set-dankclaw-bootline.sh` - [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index fee777e..f9017ee 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -387,9 +387,9 @@ The LLM inference fleet on the two hosts. Full index: and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid — its measured envelope is why the fleet went back -- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with - `amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost - (the NPU endpoint), and how to revert +- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: `iommu=pt` + since 2026-10-18 (measured cost of identity DMA: nothing — +2.0 % prefill), the + `amd_iommu=off` era it replaced, and the tracked file + installer that own it - [[kingjones-strix-rocmfp4]] — the PRIOR frank-dragoon runbook (2026-08-30 → 2026-09-11) and the rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index fa4d551..91723ff 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2066,3 +2066,45 @@ every restart since the cutover. vendor number, and the plan posted to Gitea issue #3 (comment 23, then edited). bats 371/371; commits `7643381` plus this correction. Nothing deployed — no service was touched. + +## 2026-10-18 — boot line flipped: Dankclaw now boots `iommu=pt` + +- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the research + demanded. Provenance first: `/etc/default/grub` had never been in the repo + while the IOMMU line was the most consequential kernel decision on the box, so + it is now tracked at `dankclaw/host/grub` (fetched byte-exact, 1581 B) and moves + only through `scripts/set-dankclaw-bootline.sh` — diff, install 0644 root:root, + `update-grub`, **verify the default `grub.cfg` entry**, reboot on `--reboot` + only. The deployer plans nothing under `dankclaw/host/`: kernel state is not a + container restart. +- **Measured arms, byte-identical traffic.** OFF arm at uptime 2 d 23 h: cold + prefill **1,585.1 t/s** (59,024 tok), warm 208,531, cache ratio 131.6×, needle + hit/hit @21,568, decode 37.3 t/s. PT arm after the reboot: **1,616.0 t/s + (+2.0 %)**, warm 214,131, ratio 132.5×, needle hit/hit, decode 33.7 t/s. The + corpus was rebuilt after the reboot deleted `/tmp` (tmpfs) and is provably the + same input: 2,891 lines, **1,287,656 B**, and the engine billed identical token + counts. Verdict: **identity DMA costs nothing on this host** — the 13–16 % that + justified `amd_iommu=off` for five weeks never existed here. Decode's −9.7 % is + instrument noise: three repeats of the same request on the same boot gave 31.8, + 46.9, 47.8 t/s. +- Gates after boot: 0 failed units, systemd running, 32 IOMMU groups (31 identity + + 1 DMA = the Genesys Logic SD reader `c2:00.0`, a kernel quirk), iGPU `c5:00.0` + identity group 20 with ATS, interrupt remapping on. frank-dragoon's ledger is + byte-for-byte the same shape (62.9 + 5.5 + 5.7 = 74.1 GiB, 2 slots over ONE + 200,704 pool, MTP depth 2, prompt cache ON, no clamp; listening at 39.8 s), and + Gitea / Caddy edge / Grafana answered 200. +- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`, firmware + `amdnpu/17f0_11/npu_7.sbin`, `amdxdna_accel_driver 0.7.0 … minor 0`, group 26 + `identity`, PASID width 10. `/dev/accel/accel0` `crw-rw---- root:render 261,0` + with `dank` in `render`. No endpoint deployed — revival needs a quadlet with + `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. +- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU timing for + every workload and nothing co-runs on the NPU yet) and any PerfOpt work (kernel + `7.0.0-38` has zero `perfopt` symbols; the 26.04 archive caps at `7.0.0-38.38`). +- New tool: `scripts/stage-frank-dragoon-eval.sh` rebuilds `/tmp/halogen-eval` + byte-identically (corpus + probe PNG + battery) — run it through `bash -lc`, + `uv` is in `~/.local/bin`. Docs re-keyed: decision record (banner, front matter, + verification/revert, new § *2026-10-18 execution*), `frank-npu.md` (precondition + satisfied, endpoint still absent), AGENTS.md key lesson, both index pages. + bats 371/371. frank-dragoon and frank-zealot were live the whole window; only + the reboot interrupted them (~40 s to listen, health 200).