Frank
9b372acde4
Reading patch v2 1/2 verbatim: amd_iommu_enable_perfopt() rehoming needs the device ALREADY on the identity domain and switches ATS/PRI/PASID/GCR3 off via the skip_caps path. So an iGPU in identity mode still pays a 1:1 page walk plus those four capabilities until PerfOpt exists -- iommu=pt recovers most of the gap, PerfOpt the last part. The first pass claimed pt alone already avoids the cost; that was wrong and is retracted here and in the posted issue comment (edit 23). Also re-keys frank-npu.md, which carried the dead law 'iommu=pt is not a middle ground' as its retirement rationale: identity mode + SVA is exactly the revival path (amdxdna binds its own IOPT_TYPE_SVA domain), and the cost is single-digit percent on this silicon class. Procedure step 2 now demands a local off-vs-pt measurement instead of inheriting the vendor's 13-16%, whose slow-arm boot line our record never identified.
146 lines
7.4 KiB
Markdown
146 lines
7.4 KiB
Markdown
---
|
||
type: reference
|
||
title: frank-npu (retired) — FLM NPU endpoint on Dankclaw
|
||
description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went.
|
||
tags:
|
||
- flm
|
||
- fastflowlm
|
||
- xdna2
|
||
- npu
|
||
- dankclaw
|
||
- frank-npu
|
||
- retired
|
||
timestamp: 2026-09-11
|
||
---
|
||
|
||
# frank-npu — the retired FLM NPU endpoint
|
||
|
||
> **RETIRED 2026-09-11.** The endpoint is gone by operator decision, together
|
||
> with the IOMMU configuration it required — see
|
||
> [[dankclaw-iommu-translation-off]]. The service unit, env file and serve
|
||
> script are staged in `serverAppConfigs/.AGENTTRASH/` (suffixed `.flm-npu`),
|
||
> and the host unit is stopped and masked.
|
||
|
||
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
|
||
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
|
||
whole local model registry on port 7771 and routing per request by the exact
|
||
FLM registry tag. Its one capability the iGPU fleet lacked was **sight** — and
|
||
briefly **sound**: the Gemma4 E4B IT NPU2 pack carries both a vision and an
|
||
audio tower, which is why the service ran that pack rather than the
|
||
uncensored-but-text-only alternative.
|
||
|
||
This page is kept as the retirement record. The operational detail below was
|
||
live-verified while the service ran and remains accurate about FLM's shape.
|
||
|
||
## Why it is retired
|
||
|
||
The replacement large-model runtime is worth **13–16 % of prefill** if the
|
||
kernel boots with `amd_iommu=off` instead of `iommu=pt`, measured by the
|
||
upstream project on this exact silicon. That single flag is the whole decision:
|
||
|
||
- With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand
|
||
out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device").
|
||
That is precisely the failure the endpoint was rescued from on 2026-08-30,
|
||
recorded in the next section.
|
||
- `iommu=pt` was recorded here as "not a middle ground", on the theory that the
|
||
passthrough domain still costs the vendor's prefill number. **That claim did
|
||
not survive the 2026-10-18 research.** Identity mode is not free for the iGPU
|
||
— its DMA still walks 1:1 page tables and the AMD attach path enables
|
||
ATS/PRI/PASID/GCR3 for the device — but on this silicon class the cost is
|
||
single-digit percent, and **AMD PerfOpt** (Linux 7.4, merged 2026-09-24)
|
||
removes it: the IOMMU then enforces only the IR/IW permission bits for the
|
||
privileged iGPU and stays fully armed for every other device, the NPU
|
||
included. So `pt` IS the revival path for this endpoint. See
|
||
[[dankclaw-iommu-translation-off]] § *2026-10-18 research* for the kernel
|
||
census, the versions to avoid (6.18–6.18.7 disable SVA), the device-access
|
||
quadlet law (`AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`), and the
|
||
measurement step that must precede any decision.
|
||
- The operator took the vendor's prefill figure on the September evidence, so the
|
||
NPU endpoint went.
|
||
|
||
Consequences accepted at retirement: **audio input left the fleet with no
|
||
replacement** (halogen reads images only, no audio or video), and the surviving
|
||
image path is halogen's vision tower on `frank-dragoon`.
|
||
|
||
## Where its routes went
|
||
|
||
| consumer | setting | now |
|
||
|---|---|---|
|
||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) |
|
||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` |
|
||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-dragoon` |
|
||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-zealot` |
|
||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-dragoon:medium` |
|
||
|
||
The `flm (dankclaw-npu)` provider block was removed from the harness model
|
||
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
|
||
|
||
## What it was
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
|
||
| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||
| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
|
||
| Port | `7771` on `0.0.0.0` |
|
||
| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything |
|
||
| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) |
|
||
| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` |
|
||
|
||
The serve script exported `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`, with kernels symlinked
|
||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories, so a
|
||
tag swap was an env-file edit rather than a script edit.
|
||
|
||
## API contract (as it behaved)
|
||
|
||
- The endpoint served the **whole local registry** (36 tags at first deploy) on
|
||
one port; `/v1/models` listed them all. Chat completions routed per request
|
||
by the exact registry tag in the `model` field — never a service alias.
|
||
- **Unknown tags did not 404**: `/v1/chat/completions` answered HTTP 200 with a
|
||
flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`.
|
||
Status-code-only clients silently got an empty response.
|
||
- Health was `GET /v1/version`; FLM has **no `/health` route**.
|
||
- The NPU is a single-lane device — generations serialized FIFO. It was a
|
||
background/triage endpoint, not interactive multi-user chat.
|
||
|
||
## The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)
|
||
|
||
First deploy crash-looped every 10 s with
|
||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. The chain,
|
||
all live-verified on the host:
|
||
|
||
1. `amdxdna_drm_open` failed **SVA bind with ret -19** for every client — the
|
||
XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||
2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then
|
||
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
|
||
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
|
||
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
|
||
identity-mapped (no DMA-translation overhead for `frank-dragoon`) while
|
||
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
|
||
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
|
||
zero `SVA bind device failed` entries since boot, `accel0` recreated at
|
||
boot, service active with `NRestarts=0`.
|
||
|
||
The same root cause explained the older
|
||
`amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel line — a
|
||
consequence of IOMMU-off, not a GPU fault.
|
||
|
||
**The standing rule this produced — "never set `iommu=off` on this host again" —
|
||
has been superseded**: the NPU it protected is retired by decision, and
|
||
`amd_iommu=off` is now intentional. See
|
||
[[dankclaw-iommu-translation-off]] for the current posture and how to revert it.
|
||
|
||
## Client wiring (historical)
|
||
|
||
The pi/harness model config registered this endpoint with the **exact** FLM
|
||
registry tag as the model `id` and `baseUrl` `http://192.168.0.143:7771/v1`.
|
||
Both that provider block and Hermes's `dankclaw-npu` provider were removed at
|
||
retirement; the routes are mapped in the table above.
|
||
|
||
## Citations
|
||
|
||
- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
|
||
- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
|
||
- [[halogen-flash-server]] — the surviving image-input path
|