--- type: configuration title: frank-npu Runbook description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy. tags: - flm - fastflowlm - xdna2 - npu - dankclaw - frank-npu timestamp: 2026-09-01 --- # frank-npu — FLM NPU sight+sound Runbook `frank-npu` is the third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`) running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) — which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]], `frank-large`) does not do. The model-selection tradeoff: the only NPU2 pack family on this box with BOTH a vision and an audio tower is the Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx) is text-only and fails the hard modality requirement, so this endpoint trades "uncensored" for "sight + sound". ## Deployment shape | Field | Value | |-------|-------| | Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU | | Service | `frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) | | Files | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` | | Deploy | `scripts/deploy.sh dankclaw --service frank-npu` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) | | Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) | | Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows | | Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) | | ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` | The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json` and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag swap is therefore an env-file edit, not a script edit. ## Model tags for switching FLM routes per-request by the exact registry tag in the `model` field. To swap the service model, change `FLM_MODEL` in the env file and restart. To let clients choose, send requests with the desired `model` field. | Tag | Model | Modality | Size | Family | |-----|-------|----------|------|--------| | `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 | | `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e | | `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 | | `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss | **To swap the active model** (changes what the service loads at start): ``` sed -i 's/^FLM_MODEL=.*/FLM_MODEL=/' scripts/frank-npu.env ./scripts/deploy.sh dankclaw --service frank-npu ``` **To switch per-request** (service stays loaded, client sends the tag): ``` curl http://192.168.0.143:7771/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}' ``` Known tags are in the user registry at `/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with `{"error":"Failed to load model!"}` (no choices). ## Serving model and API contract - The endpoint serves the **whole local registry** (36 tags at first deploy) on one port; `/v1/models` lists them all. Chat completions route per request by the exact FLM registry tag in the `model` field — never a service alias. - **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions` answers HTTP 200 with a flat error body `{"error":"Failed to load model!"}` and no `choices`; `/v1/completions` answers an OpenAI-shaped nested error (`type: server_error`, `code: 500`), also under HTTP 200. A valid tag routes that request to that model (response echoes the requested tag); no server restart is needed to switch between registry tags. Status-code-only clients silently get an empty response — check for the `error` key. - Multimodal content parts accepted by `/v1/chat/completions`: `{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and `{"type":"input_audio","input_audio":{"data":"","format":"wav"}}`. - Chat template roles: only `system`/`user`/`assistant`; any other role fails the request with `Unexpected message role.`. - The NPU is a single-lane device: generations serialize in FIFO order. This is a background/triage endpoint, not an interactive multi-user chat. ## Live-verified behavior (2026-08-30, first deploy) - Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was answered correctly ("The image is a solid red color."); the request consumed 279 prompt tokens — the vision tower consumed the pixels (a text-only model given zero image bytes cannot answer "what color"). - Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed 37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower tokens; the model answered truthfully that it heard no speech (it is a tone, not speech — ASR stays off). ## The IOMMU/SVA root cause (first deploy crashed, fixed same day) First deploy crash-looped every 10 s (`RestartSec=10`) with `Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause chain, all live-verified on the host: 1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client — the XDNA2 driver cannot hand out the accel device without PASID/SVA. 2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried `iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the 2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`. 3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT` on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the iGPU default domain identity-mapped (zero DMA-translation overhead for `frank-large`) while enabling IOMMU hardware + PASID, which is what the NPU driver's SVA bind requires — PASID domains are paged regardless of the default-domain type. 4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups), zero `SVA bind device failed` journal entries since boot, accel0 recreated at boot, service active with `NRestarts=0`, and the image/audio probes above served live. The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU fault. Do not set `iommu=off` on this host again; it silently kills the NPU. ## Client wiring The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware) registers this endpoint under the Dankclaw provider with the **exact** FLM registry tag as the model `id` (unknown tags answer a 200-wrapped error, not a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context window equal to the serving ctx-len. Health checks for this service must use `GET /v1/version` (there is no `/health` route).