serverAppConfigs/serverAppConfigs-docs/LLM/frank-npu.md

---
type: configuration
title: frank-default-sightsound Runbook
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.
tags:
  - flm
  - fastflowlm
  - xdna2
  - npu
  - dankclaw
  - frank-default-sightsound
timestamp: 2026-09-01
---

# frank-default-sightsound — FLM NPU sight+sound Runbook

`frank-default-sightsound` is the third inference endpoint on Dankclaw: a
FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`)
running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) —
which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]],
`frank-default-large`) does not do. The model-selection tradeoff: the only
NPU2 pack family on this box with BOTH a vision and an audio tower is the
Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx)
is text-only and fails the hard modality requirement, so this endpoint trades
"uncensored" for "sight + sound".

## Deployment shape

| Field | Value |
|-------|-------|
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` |
| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) |
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) |
| ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |

The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag
swap is therefore an env-file edit, not a script edit.

## Model tags for switching

FLM routes per-request by the exact registry tag in the `model` field. To swap
the service model, change `FLM_MODEL` in the env file and restart. To let
clients choose, send requests with the desired `model` field.

| Tag | Model | Modality | Size | Family |
|-----|-------|----------|------|--------|
| `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 |
| `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e |
| `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 |
| `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss |

**To swap the active model** (changes what the service loads at start):
```
sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-default-sightsound.env
./scripts/deploy.sh dankclaw --service frank-default-sightsound
```

**To switch per-request** (service stays loaded, client sends the tag):
```
curl http://192.168.0.143:7771/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}'
```

Known tags are in the user registry at
`/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with
`{"error":"Failed to load <tag> model!"}` (no choices).

## Serving model and API contract

- The endpoint serves the **whole local registry** (36 tags at first deploy) on
  one port; `/v1/models` lists them all. Chat completions route per request by
  the exact FLM registry tag in the `model` field — never a service alias.
- **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions`
  answers HTTP 200 with a flat error body
  `{"error":"Failed to load <tag> model!"}` and no `choices`;
  `/v1/completions` answers an OpenAI-shaped nested error
  (`type: server_error`, `code: 500`), also under HTTP 200. A valid tag
  routes that request to that model (response echoes the requested tag); no
  server restart is needed to switch between registry tags. Status-code-only
  clients silently get an empty response — check for the `error` key.
- Multimodal content parts accepted by `/v1/chat/completions`:
  `{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and
  `{"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}`.
- Chat template roles: only `system`/`user`/`assistant`; any other role fails
  the request with `Unexpected message role.`.
- The NPU is a single-lane device: generations serialize in FIFO order. This is
  a background/triage endpoint, not an interactive multi-user chat.

## Live-verified behavior (2026-08-30, first deploy)

- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was
  answered correctly ("The image is a solid red color."); the request consumed
  279 prompt tokens — the vision tower consumed the pixels (a text-only model
  given zero image bytes cannot answer "what color").
- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed
  37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower
  tokens; the model answered truthfully that it heard no speech (it is a tone,
  not speech — ASR stays off).

## The IOMMU/SVA root cause (first deploy crashed, fixed same day)

First deploy crash-looped every 10 s (`RestartSec=10`) with
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause
chain, all live-verified on the host:

1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client —
   the XDNA2 driver cannot hand out the accel device without PASID/SVA.
2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried
   `iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the
   2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID
   state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`.
3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT`
   on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the
   iGPU default domain identity-mapped (zero DMA-translation overhead for
   `frank-default-large`) while enabling IOMMU hardware + PASID, which is what
   the NPU driver's SVA bind requires — PASID domains are paged regardless of
   the default-domain type.
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
   zero `SVA bind device failed` journal entries since boot, accel0 recreated at
   boot, service active with `NRestarts=0`, and the image/audio probes above
   served live.

The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid
mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU
fault. Do not set `iommu=off` on this host again; it silently kills the NPU.

## Client wiring

The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
registers this endpoint under the Dankclaw provider with the **exact** FLM
registry tag as the model `id` (unknown tags answer a 200-wrapped error, not
a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context
window equal to the serving ctx-len. Health checks for this service must use
`GET /v1/version` (there is no `/health` route).