serverAppConfigs/serverAppConfigs-docs/LLM/frank-npu.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-npu Runbook	FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.	flm fastflowlm xdna2 npu dankclaw frank-npu	2026-09-01

frank-npu — FLM NPU sight+sound Runbook

frank-npu is the third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (/usr/bin/flm, dpkg package fastflowlm) running on the XDNA2 NPU (50 TOPS). It serves sight (vision) — which the iGPU llama.cpp fleet (kingjones-strix-rocmfp4, frank-large) does not do. The model-selection tradeoff: the only NPU2 pack family on this box with BOTH a vision and an audio tower is the Gemma4 E4B IT pack; the uncensored candidate (qwythos:9b, 8.94 GB, 262k ctx) is text-only and fails the hard modality requirement, so this endpoint trades "uncensored" for "sight + sound".

Deployment shape

Field	Value
Runtime	FLM v1.0.1 (/usr/bin/flm, dpkg fastflowlm) on the XDNA2 NPU
Service	frank-npu.service (systemd system unit, User=dank, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity)
Files	scripts/frank-npu.env, scripts/serve-frank-npu.sh, scripts/frank-npu.service
Deploy	scripts/deploy.sh dankclaw --service frank-npu (relink + daemon-reload + enable + restart; health gate polls /v1/version, 300 s cap — FLM has no /health route)
Port	7771 on 0.0.0.0 (52625/52626 left free for ad-hoc flm serve experiments)
Power mode	FLM_PMODE=turbo — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to performance if iGPU power-budget contention ever shows
Context	`FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.)
ASR	FLM_ASR=0 — the E4B pack answers images natively through its vision tower in /v1/chat/completions; setting FLM_ASR=1 additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes /v1/audio/transcriptions

The serve script exports FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json and FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins; kernels are symlinked from /opt/fastflowlm/share/flm/xclbins/ into the per-pack directories. A tag swap is therefore an env-file edit, not a script edit.

Model tags for switching

FLM routes per-request by the exact registry tag in the model field. To swap the service model, change FLM_MODEL in the env file and restart. To let clients choose, send requests with the desired model field.

Tag	Model	Modality	Size	Family
qwen3.8-distilled-heretic:9b	Qwen3.8 Distilled Heretic	vision + reasoning	7.6 GB	qwen3.5
gemma4-it:e4b	Gemma4 E4B IT	vision + audio	9.1 GB	gemma4e
qwythos:9b	Qwythos 9B Claude Mythos	vision + reasoning	8.9 GB	qwen3.5
gpt-oss:20b	GPT-OSS 20B	text only	13.8 GB	gpt-oss

To swap the active model (changes what the service loads at start):

sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-npu.env
./scripts/deploy.sh dankclaw --service frank-npu

To switch per-request (service stays loaded, client sends the tag):

curl http://192.168.0.143:7771/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}'

Known tags are in the user registry at /home/dank/.config/flm/model_list.json; unknown tags answer HTTP 200 with {"error":"Failed to load <tag> model!"} (no choices).

Serving model and API contract

• The endpoint serves the whole local registry (36 tags at first deploy) on one port; /v1/models lists them all. Chat completions route per request by the exact FLM registry tag in the model field — never a service alias.
• Unknown model tags do NOT 404 (live-probed): /v1/chat/completions answers HTTP 200 with a flat error body {"error":"Failed to load <tag> model!"} and no choices; /v1/completions answers an OpenAI-shaped nested error (type: server_error, code: 500), also under HTTP 200. A valid tag routes that request to that model (response echoes the requested tag); no server restart is needed to switch between registry tags. Status-code-only clients silently get an empty response — check for the error key.
• Multimodal content parts accepted by /v1/chat/completions: {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}} and {"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}.
• Chat template roles: only system/user/assistant; any other role fails the request with Unexpected message role..
• The NPU is a single-lane device: generations serialize in FIFO order. This is a background/triage endpoint, not an interactive multi-user chat.

Live-verified behavior (2026-08-30, first deploy)

• Vision tower: a generated red 8×8 PNG posted as an image_url data-URI was answered correctly ("The image is a solid red color."); the request consumed 279 prompt tokens — the vision tower consumed the pixels (a text-only model given zero image bytes cannot answer "what color").
• Audio tower: a 440 Hz decaying sine WAV posted as input_audio consumed 37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower tokens; the model answered truthfully that it heard no speech (it is a tone, not speech — ASR stays off).

The IOMMU/SVA root cause (first deploy crashed, fixed same day)

First deploy crash-looped every 10 s (RestartSec=10) with Error: Open /dev/accel/accel0 failed (err=-19): No such device. Root-cause chain, all live-verified on the host:

1. amdxdna_drm_open was failing SVA bind with ret -19 for every client — the XDNA2 driver cannot hand out the accel device without PASID/SVA.
2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried iommu=off (a perf/DMA tuning experiment from 2026-08-20, applied at the 2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID state; PASIDCtl: Enable- was observed disabled on 0000:c5:00.1.
3. Fix: replace iommu=off with iommu=pt in GRUB_CMDLINE_LINUX_DEFAULT on /etc/default/grub, update-grub, reboot. Passthrough mode keeps the iGPU default domain identity-mapped (zero DMA-translation overhead for frank-large) while enabling IOMMU hardware + PASID, which is what the NPU driver's SVA bind requires — PASID domains are paged regardless of the default-domain type.
4. Post-boot verification: /sys/kernel/iommu_groups/ populated (30 groups), zero SVA bind device failed journal entries since boot, accel0 recreated at boot, service active with NRestarts=0, and the image/audio probes above served live.

The same root cause explains the older amdgpu 0000:c4:00.0: no vmid pasid mapping supported kernel log line — a consequence of IOMMU-off, not a GPU fault. Do not set iommu=off on this host again; it silently kills the NPU.

Client wiring

The pi client model config (/home/dank/.pi/agent/models.json on Dankware) registers this endpoint under the Dankclaw provider with the exact FLM registry tag as the model id (unknown tags answer a 200-wrapped error, not a silent 1B fallback), baseUrl http://192.168.0.143:7771/v1, and a per-request context window equal to the serving ctx-len. Health checks for this service must use GET /v1/version (there is no /health route).