frank-default-sightsound — FLM NPU sight+sound Runbook

frank-default-sightsound is the third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (/usr/bin/flm, dpkg package fastflowlm) running on the XDNA2 NPU (50 TOPS). It serves sight (vision) — which the iGPU llama.cpp fleet (kingjones-strix-rocmfp4, frank-default-large) does not do. The model-selection tradeoff: the only NPU2 pack family on this box with BOTH a vision and an audio tower is the Gemma4 E4B IT pack; the uncensored candidate (qwythos:9b, 8.94 GB, 262k ctx) is text-only and fails the hard modality requirement, so this endpoint trades "uncensored" for "sight + sound".

Deployment shape

Field	Value
Runtime	FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU
Service	`frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`)
Files	`scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service`
Deploy	`scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has no `/health` route)
Port	`7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments)
Power mode	`FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows
Context	`FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.)
ASR	`FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions`

The serve script exports FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json and FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins; kernels are symlinked from /opt/fastflowlm/share/flm/xclbins/ into the per-pack directories. A tag swap is therefore an env-file edit, not a script edit.

Model tags for switching

FLM routes per-request by the exact registry tag in the model field. To swap the service model, change FLM_MODEL in the env file and restart. To let clients choose, send requests with the desired model field.

Tag	Model	Modality	Size	Family
`qwen3.8-distilled-heretic:9b`	Qwen3.8 Distilled Heretic	vision + reasoning	7.6 GB	qwen3.5
`gemma4-it:e4b`	Gemma4 E4B IT	vision + audio	9.1 GB	gemma4e
`qwythos:9b`	Qwythos 9B Claude Mythos	vision + reasoning	8.9 GB	qwen3.5
`gpt-oss:20b`	GPT-OSS 20B	text only	13.8 GB	gpt-oss

To swap the active model (changes what the service loads at start):

sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-default-sightsound.env
./scripts/deploy.sh dankclaw --service frank-default-sightsound

To switch per-request (service stays loaded, client sends the tag):

curl http://192.168.0.143:7771/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}'

Known tags are in the user registry at /home/dank/.config/flm/model_list.json; unknown tags answer HTTP 200 with {"error":"Failed to load <tag> model!"} (no choices).

Serving model and API contract

The endpoint serves the whole local registry (36 tags at first deploy) on one port; /v1/models lists them all. Chat completions route per request by the exact FLM registry tag in the model field — never a service alias.
Unknown model tags do NOT 404 (live-probed): /v1/chat/completions answers HTTP 200 with a flat error body {"error":"Failed to load <tag> model!"} and no choices; /v1/completions answers an OpenAI-shaped nested error (type: server_error, code: 500), also under HTTP 200. A valid tag routes that request to that model (response echoes the requested tag); no server restart is needed to switch between registry tags. Status-code-only clients silently get an empty response — check for the error key.
Multimodal content parts accepted by /v1/chat/completions: {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}} and {"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}.
Chat template roles: only system/user/assistant; any other role fails the request with Unexpected message role..
The NPU is a single-lane device: generations serialize in FIFO order. This is a background/triage endpoint, not an interactive multi-user chat.

Live-verified behavior (2026-08-30, first deploy)

Vision tower: a generated red 8×8 PNG posted as an image_url data-URI was answered correctly ("The image is a solid red color."); the request consumed 279 prompt tokens — the vision tower consumed the pixels (a text-only model given zero image bytes cannot answer "what color").
Audio tower: a 440 Hz decaying sine WAV posted as input_audio consumed 37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower tokens; the model answered truthfully that it heard no speech (it is a tone, not speech — ASR stays off).

The IOMMU/SVA root cause (first deploy crashed, fixed same day)

First deploy crash-looped every 10 s (RestartSec=10) with Error: Open /dev/accel/accel0 failed (err=-19): No such device. Root-cause chain, all live-verified on the host:

amdxdna_drm_open was failing SVA bind with ret -19 for every client — the XDNA2 driver cannot hand out the accel device without PASID/SVA.
SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried iommu=off (a perf/DMA tuning experiment from 2026-08-20, applied at the 2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID state; PASIDCtl: Enable- was observed disabled on 0000:c5:00.1.
Fix: replace iommu=off with iommu=pt in GRUB_CMDLINE_LINUX_DEFAULT on /etc/default/grub, update-grub, reboot. Passthrough mode keeps the iGPU default domain identity-mapped (zero DMA-translation overhead for frank-default-large) while enabling IOMMU hardware + PASID, which is what the NPU driver's SVA bind requires — PASID domains are paged regardless of the default-domain type.
Post-boot verification: /sys/kernel/iommu_groups/ populated (30 groups), zero SVA bind device failed journal entries since boot, accel0 recreated at boot, service active with NRestarts=0, and the image/audio probes above served live.

The same root cause explains the older amdgpu 0000:c4:00.0: no vmid pasid mapping supported kernel log line — a consequence of IOMMU-off, not a GPU fault. Do not set iommu=off on this host again; it silently kills the NPU.

Client wiring

The pi client model config (/home/dank/.pi/agent/models.json on Dankware) registers this endpoint under the Dankclaw provider with the exact FLM registry tag as the model id (unknown tags answer a 200-wrapped error, not a silent 1B fallback), baseUrl http://192.168.0.143:7771/v1, and a per-request context window equal to the serving ctx-len. Health checks for this service must use GET /v1/version (there is no /health route).

7.7 KiB Raw Blame History Unescape Escape