docs(sightsound): NPU sight+sound runbook, fleet table, IOMMU/SVA root cause
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every /dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot. Whole-registry single-port serving with per-request routing by exact FLM tag; unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version. Vision + audio towers proven live (image_url color probe; audio control-pair token-count probe). Updates AGENTS.md fleet table + runtime section, both index pages, and the change log.
This commit is contained in:
Frank
@@ -0,0 +1,106 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-default-sightsound Runbook
|
||||
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving the Gemma4 E4B vision+audio pack on port 52626, including the IOMMU/SVA root cause that took it down at first deploy.
|
||||
tags:
|
||||
- flm
|
||||
- fastflowlm
|
||||
- xdna2
|
||||
- npu
|
||||
- dankclaw
|
||||
- frank-default-sightsound
|
||||
timestamp: 2026-08-30
|
||||
---
|
||||
|
||||
# frank-default-sightsound — FLM NPU sight+sound Runbook
|
||||
|
||||
`frank-default-sightsound` is the third inference endpoint on Dankclaw: a
|
||||
FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`)
|
||||
running on the **XDNA2 NPU** (50 TOPS). It exists to serve *sight and sound* —
|
||||
image and audio input — which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]],
|
||||
`frank-default-large`) does not do. The model-selection tradeoff: the only
|
||||
NPU2 pack family on this box with BOTH a vision and an audio tower is the
|
||||
Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx)
|
||||
is text-only and fails the hard modality requirement, so this endpoint trades
|
||||
"uncensored" for "sight + sound".
|
||||
|
||||
## Deployment shape
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
|
||||
| Service | `frank-default-sightsound.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||
| Files | `scripts/frank-default-sightsound.env`, `scripts/serve-frank-default-sightsound.sh`, `scripts/frank-default-sightsound.service` |
|
||||
| Deploy | `scripts/deploy.sh dankclaw --service frank-default-sightsound` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
|
||||
| Port | `52626` on `0.0.0.0` (52625 left free for ad-hoc `flm serve` experiments) |
|
||||
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
|
||||
| Context | `FLM_CTX_LEN` = verified max for the E4B pack on this box (native 128K context; the registry's default context length is only the fallback). FLM rounds non-power-of-2 values up. |
|
||||
| ASR | `FLM_ASR=0` — the E4B pack answers audio natively through its own audio tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |
|
||||
|
||||
The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked
|
||||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag
|
||||
swap is therefore an env-file edit, not a script edit.
|
||||
|
||||
## Serving model and API contract
|
||||
|
||||
- The endpoint serves the **whole local registry** (36 tags at first deploy) on
|
||||
one port; `/v1/models` lists them all. Chat completions route per request by
|
||||
the exact FLM registry tag in the `model` field — never a service alias.
|
||||
- **Unknown model tags do NOT 404**: FLM logs an error and silently serves
|
||||
`llama3.2:1b` instead. Client configs must carry the exact registry tag or
|
||||
they will silently get the 1B fallback.
|
||||
- Multimodal content parts accepted by `/v1/chat/completions`:
|
||||
`{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and
|
||||
`{"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}`.
|
||||
- Chat template roles: only `system`/`user`/`assistant`; any other role fails
|
||||
the request with `Unexpected message role.`.
|
||||
- The NPU is a single-lane device: generations serialize in FIFO order. This is
|
||||
a background/triage endpoint, not an interactive multi-user chat.
|
||||
|
||||
## Live-verified behavior (2026-08-30, first deploy)
|
||||
|
||||
- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was
|
||||
answered correctly ("The image is a solid red color."); the request consumed
|
||||
279 prompt tokens — the vision tower consumed the pixels (a text-only model
|
||||
given zero image bytes cannot answer "what color").
|
||||
- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed
|
||||
37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower
|
||||
tokens; the model answered truthfully that it heard no speech (it is a tone,
|
||||
not speech — ASR stays off).
|
||||
|
||||
## The IOMMU/SVA root cause (first deploy crashed, fixed same day)
|
||||
|
||||
First deploy crash-looped every 10 s (`RestartSec=10`) with
|
||||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause
|
||||
chain, all live-verified on the host:
|
||||
|
||||
1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client —
|
||||
the XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||||
2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried
|
||||
`iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the
|
||||
2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`.
|
||||
3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT`
|
||||
on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the
|
||||
iGPU default domain identity-mapped (zero DMA-translation overhead for
|
||||
`frank-default-large`) while enabling IOMMU hardware + PASID, which is what
|
||||
the NPU driver's SVA bind requires — PASID domains are paged regardless of
|
||||
the default-domain type.
|
||||
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
|
||||
zero `SVA bind device failed` journal entries since boot, accel0 recreated at
|
||||
boot, service active with `NRestarts=0`, and the image/audio probes above
|
||||
served live.
|
||||
|
||||
The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid
|
||||
mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU
|
||||
fault. Do not set `iommu=off` on this host again; it silently kills the NPU.
|
||||
|
||||
## Client wiring
|
||||
|
||||
The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
|
||||
registers this endpoint under the Dankclaw provider with the **exact** FLM
|
||||
registry tag as the model `id` (see above: unknown tags silently fall back to
|
||||
the 1B), `baseUrl` `http://192.168.0.143:52626/v1`, and a per-request context
|
||||
window equal to the serving ctx-len. Health checks for this service must use
|
||||
`GET /v1/version` (there is no `/health` route).
|
||||
@@ -78,6 +78,10 @@ ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on
|
||||
Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port
|
||||
52626, whole-registry serving, per-request model routing), incl. the
|
||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
|
||||
@@ -127,6 +127,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
||||
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
||||
- [[frank-default-sightsound]] — ACTIVE FLM NPU sight+sound runbook on Dankclaw:
|
||||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 52626; whole local
|
||||
registry served on one port with per-request model routing; includes the
|
||||
IOMMU/SVA root cause and the `iommu=pt` fix behind its first-deploy crash
|
||||
|
||||
### Benchmarks and Sizing
|
||||
|
||||
|
||||
@@ -11,6 +11,38 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-08-30 (night II) — frank-default-sightsound deployed; NPU SVA/IOMMU root cause
|
||||
- **Deployed** `frank-default-sightsound` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`,
|
||||
dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626;
|
||||
primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy
|
||||
crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`.
|
||||
- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19
|
||||
whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a
|
||||
perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU
|
||||
groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The
|
||||
same root cause explains the `amdgpu ... no vmid pasid mapping supported`
|
||||
kernel line — a consequence, not a GPU fault.
|
||||
- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps
|
||||
the iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||||
frank-default-large — while enabling IOMMU hardware + PASID, which SVA needs;
|
||||
PASID domains are paged regardless of the default-domain type). Post-boot:
|
||||
30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`,
|
||||
`NRestarts=0` since boot.
|
||||
- **Live-verified behavior**: whole-registry serving with per-request routing by
|
||||
exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved
|
||||
end-to-end (generated red PNG as `image_url` data-URI → correct color answer,
|
||||
279 prompt tokens consumed); audio tower proven with a control pair (440 Hz
|
||||
tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only
|
||||
control). Health probe is `/v1/version` — FLM has no `/health` route.
|
||||
- `/scripts/frank-default-sightsound.env`, `/scripts/serve-frank-default-sightsound.sh`,
|
||||
`/scripts/frank-default-sightsound.service` + deployer allowlist entries
|
||||
(`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent
|
||||
(`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`,
|
||||
`LimitMEMLOCK=infinity` — FLM requires unlimited memlock).
|
||||
- **Docs**: new runbook [[frank-default-sightsound]] (linked from both indexes),
|
||||
AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard
|
||||
requirement. Client wiring: pi `models.json` registers the endpoint under the
|
||||
Dankclaw provider with the exact FLM registry tag as model id.
|
||||
|
||||
## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
|
||||
- **Dankclaw / frank-default-large**: swapped to
|
||||
|
||||
Reference in New Issue
Block a user