feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
This commit is contained in:
Frank committed
1 parent
184df9aa0d
commit
301914b1c0
21 files changed
+929
-279
No files matched your search
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-npu Runbook
|
||||
description: FastFlowLM (FLM) NPU sight+sound endpoint on Dankclaw — XDNA2 NPU serving Qwen3.8 Distilled Heretic vision+text pack on port 7771, including the IOMMU/SVA root cause that took it down at first deploy.
|
||||
type: reference
|
||||
title: frank-npu (retired) — FLM NPU endpoint on Dankclaw
|
||||
description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went.
|
||||
tags:
|
||||
- flm
|
||||
- fastflowlm
|
||||
@@ -9,133 +9,125 @@ tags:
|
||||
- npu
|
||||
- dankclaw
|
||||
- frank-npu
|
||||
timestamp: 2026-09-01
|
||||
- retired
|
||||
timestamp: 2026-09-11
|
||||
---
|
||||
|
||||
# frank-npu — FLM NPU sight+sound Runbook
|
||||
# frank-npu — the retired FLM NPU endpoint
|
||||
|
||||
`frank-npu` is the third inference endpoint on Dankclaw: a
|
||||
FastFlowLM (FLM) v1.0.1 server (`/usr/bin/flm`, dpkg package `fastflowlm`)
|
||||
running on the **XDNA2 NPU** (50 TOPS). It serves *sight* (vision) —
|
||||
which the iGPU llama.cpp fleet ([[kingjones-strix-rocmfp4]],
|
||||
`frank-large`) does not do. The model-selection tradeoff: the only
|
||||
NPU2 pack family on this box with BOTH a vision and an audio tower is the
|
||||
Gemma4 E4B IT pack; the uncensored candidate (`qwythos:9b`, 8.94 GB, 262k ctx)
|
||||
is text-only and fails the hard modality requirement, so this endpoint trades
|
||||
"uncensored" for "sight + sound".
|
||||
> **RETIRED 2026-09-11.** The endpoint is gone by operator decision, together
|
||||
> with the IOMMU configuration it required — see
|
||||
> [[dankclaw-iommu-translation-off]]. The service unit, env file and serve
|
||||
> script are staged in `serverAppConfigs/.AGENTTRASH/` (suffixed `.flm-npu`),
|
||||
> and the host unit is stopped and masked.
|
||||
|
||||
## Deployment shape
|
||||
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
|
||||
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
|
||||
whole local model registry on port 7771 and routing per request by the exact
|
||||
FLM registry tag. Its one capability the iGPU fleet lacked was **sight** — and
|
||||
briefly **sound**: the Gemma4 E4B IT NPU2 pack carries both a vision and an
|
||||
audio tower, which is why the service ran that pack rather than the
|
||||
uncensored-but-text-only alternative.
|
||||
|
||||
This page is kept as the retirement record. The operational detail below was
|
||||
live-verified while the service ran and remains accurate about FLM's shape.
|
||||
|
||||
## Why it is retired
|
||||
|
||||
The replacement large-model runtime is worth **13–16 % of prefill** if the
|
||||
kernel boots with `amd_iommu=off` instead of `iommu=pt`, measured by the
|
||||
upstream project on this exact silicon. That single flag is the whole decision:
|
||||
|
||||
- With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand
|
||||
out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device").
|
||||
That is precisely the failure the endpoint was rescued from on 2026-08-30,
|
||||
recorded in the next section.
|
||||
- `iommu=pt` is not a middle ground for the NPU *and* the prefill number: the
|
||||
passthrough domain still costs the prefill throughput the vendor measured.
|
||||
- The operator chose prefill, so the NPU endpoint goes.
|
||||
|
||||
Consequences accepted at retirement: **audio input left the fleet with no
|
||||
replacement** (halogen reads images only, no audio or video), and the surviving
|
||||
image path is halogen's vision tower on `frank-large`.
|
||||
|
||||
## Where its routes went
|
||||
|
||||
| consumer | setting | now |
|
||||
|---|---|---|
|
||||
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-large` (halogen vision tower) |
|
||||
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-small` |
|
||||
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-large` |
|
||||
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-small` |
|
||||
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-large:medium` |
|
||||
|
||||
The `flm (dankclaw-npu)` provider block was removed from the harness model
|
||||
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
|
||||
|
||||
## What it was
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
|---|---|
|
||||
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
|
||||
| Service | `frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||
| Files | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
|
||||
| Deploy | `scripts/deploy.sh dankclaw --service frank-npu` (relink + daemon-reload + enable + restart; health gate polls `/v1/version`, 300 s cap — FLM has **no `/health` route**) |
|
||||
| Port | `7771` on `0.0.0.0` (52625/52626 left free for ad-hoc `flm serve` experiments) |
|
||||
| Power mode | `FLM_PMODE=turbo` — no other tenant uses the NPU on this host (the large model runs on the iGPU), so the top power mode costs no neighbour anything; drop to `performance` if iGPU power-budget contention ever shows |
|
||||
| Context | `FLM_CTX_LEN=131072 (128K; verified max on this box; native context length.) |
|
||||
| ASR | `FLM_ASR=0` — the E4B pack answers images natively through its vision tower in `/v1/chat/completions`; setting `FLM_ASR=1` additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes `/v1/audio/transcriptions` |
|
||||
| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
|
||||
| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
|
||||
| Port | `7771` on `0.0.0.0` |
|
||||
| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything |
|
||||
| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) |
|
||||
| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` |
|
||||
|
||||
The serve script exports `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`; kernels are symlinked
|
||||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories. A tag
|
||||
swap is therefore an env-file edit, not a script edit.
|
||||
The serve script exported `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
|
||||
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`, with kernels symlinked
|
||||
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories, so a
|
||||
tag swap was an env-file edit rather than a script edit.
|
||||
|
||||
## Model tags for switching
|
||||
## API contract (as it behaved)
|
||||
|
||||
FLM routes per-request by the exact registry tag in the `model` field. To swap
|
||||
the service model, change `FLM_MODEL` in the env file and restart. To let
|
||||
clients choose, send requests with the desired `model` field.
|
||||
- The endpoint served the **whole local registry** (36 tags at first deploy) on
|
||||
one port; `/v1/models` listed them all. Chat completions routed per request
|
||||
by the exact registry tag in the `model` field — never a service alias.
|
||||
- **Unknown tags did not 404**: `/v1/chat/completions` answered HTTP 200 with a
|
||||
flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`.
|
||||
Status-code-only clients silently got an empty response.
|
||||
- Health was `GET /v1/version`; FLM has **no `/health` route**.
|
||||
- The NPU is a single-lane device — generations serialized FIFO. It was a
|
||||
background/triage endpoint, not interactive multi-user chat.
|
||||
|
||||
| Tag | Model | Modality | Size | Family |
|
||||
|-----|-------|----------|------|--------|
|
||||
| `qwen3.8-distilled-heretic:9b` | Qwen3.8 Distilled Heretic | vision + reasoning | 7.6 GB | qwen3.5 |
|
||||
| `gemma4-it:e4b` | Gemma4 E4B IT | vision + audio | 9.1 GB | gemma4e |
|
||||
| `qwythos:9b` | Qwythos 9B Claude Mythos | vision + reasoning | 8.9 GB | qwen3.5 |
|
||||
| `gpt-oss:20b` | GPT-OSS 20B | text only | 13.8 GB | gpt-oss |
|
||||
## The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)
|
||||
|
||||
**To swap the active model** (changes what the service loads at start):
|
||||
```
|
||||
sed -i 's/^FLM_MODEL=.*/FLM_MODEL=<tag>/' scripts/frank-npu.env
|
||||
./scripts/deploy.sh dankclaw --service frank-npu
|
||||
```
|
||||
First deploy crash-looped every 10 s with
|
||||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. The chain,
|
||||
all live-verified on the host:
|
||||
|
||||
**To switch per-request** (service stays loaded, client sends the tag):
|
||||
```
|
||||
curl http://192.168.0.143:7771/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"gemma4-it:e4b","messages":[...],"max_tokens":64}'
|
||||
```
|
||||
|
||||
Known tags are in the user registry at
|
||||
`/home/dank/.config/flm/model_list.json`; unknown tags answer HTTP 200 with
|
||||
`{"error":"Failed to load <tag> model!"}` (no choices).
|
||||
|
||||
## Serving model and API contract
|
||||
|
||||
- The endpoint serves the **whole local registry** (36 tags at first deploy) on
|
||||
one port; `/v1/models` lists them all. Chat completions route per request by
|
||||
the exact FLM registry tag in the `model` field — never a service alias.
|
||||
- **Unknown model tags do NOT 404** (live-probed): `/v1/chat/completions`
|
||||
answers HTTP 200 with a flat error body
|
||||
`{"error":"Failed to load <tag> model!"}` and no `choices`;
|
||||
`/v1/completions` answers an OpenAI-shaped nested error
|
||||
(`type: server_error`, `code: 500`), also under HTTP 200. A valid tag
|
||||
routes that request to that model (response echoes the requested tag); no
|
||||
server restart is needed to switch between registry tags. Status-code-only
|
||||
clients silently get an empty response — check for the `error` key.
|
||||
- Multimodal content parts accepted by `/v1/chat/completions`:
|
||||
`{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}` and
|
||||
`{"type":"input_audio","input_audio":{"data":"<base64 WAV, no data: prefix>","format":"wav"}}`.
|
||||
- Chat template roles: only `system`/`user`/`assistant`; any other role fails
|
||||
the request with `Unexpected message role.`.
|
||||
- The NPU is a single-lane device: generations serialize in FIFO order. This is
|
||||
a background/triage endpoint, not an interactive multi-user chat.
|
||||
|
||||
## Live-verified behavior (2026-08-30, first deploy)
|
||||
|
||||
- Vision tower: a generated red 8×8 PNG posted as an `image_url` data-URI was
|
||||
answered correctly ("The image is a solid red color."); the request consumed
|
||||
279 prompt tokens — the vision tower consumed the pixels (a text-only model
|
||||
given zero image bytes cannot answer "what color").
|
||||
- Audio tower: a 440 Hz decaying sine WAV posted as `input_audio` consumed
|
||||
37 prompt tokens vs 15 for the identical text-only control → ~22 audio-tower
|
||||
tokens; the model answered truthfully that it heard no speech (it is a tone,
|
||||
not speech — ASR stays off).
|
||||
|
||||
## The IOMMU/SVA root cause (first deploy crashed, fixed same day)
|
||||
|
||||
First deploy crash-looped every 10 s (`RestartSec=10`) with
|
||||
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. Root-cause
|
||||
chain, all live-verified on the host:
|
||||
|
||||
1. `amdxdna_drm_open` was failing **SVA bind with ret -19** for every client —
|
||||
the XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||||
2. SVA requires the AMD IOMMU to be enabled; the kernel cmdline carried
|
||||
`iommu=off` (a perf/DMA tuning experiment from 2026-08-20, applied at the
|
||||
2026-08-26 reboot). With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed disabled on `0000:c5:00.1`.
|
||||
3. Fix: replace `iommu=off` with **`iommu=pt`** in `GRUB_CMDLINE_LINUX_DEFAULT`
|
||||
on `/etc/default/grub`, `update-grub`, reboot. Passthrough mode keeps the
|
||||
iGPU default domain identity-mapped (zero DMA-translation overhead for
|
||||
`frank-large`) while enabling IOMMU hardware + PASID, which is what
|
||||
the NPU driver's SVA bind requires — PASID domains are paged regardless of
|
||||
the default-domain type.
|
||||
1. `amdxdna_drm_open` failed **SVA bind with ret -19** for every client — the
|
||||
XDNA2 driver cannot hand out the accel device without PASID/SVA.
|
||||
2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then
|
||||
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
|
||||
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
|
||||
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
|
||||
identity-mapped (no DMA-translation overhead for `frank-large`) while
|
||||
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
|
||||
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
|
||||
zero `SVA bind device failed` journal entries since boot, accel0 recreated at
|
||||
boot, service active with `NRestarts=0`, and the image/audio probes above
|
||||
served live.
|
||||
zero `SVA bind device failed` entries since boot, `accel0` recreated at
|
||||
boot, service active with `NRestarts=0`.
|
||||
|
||||
The same root cause explains the older `amdgpu 0000:c4:00.0: no vmid pasid
|
||||
mapping supported` kernel log line — a consequence of IOMMU-off, not a GPU
|
||||
fault. Do not set `iommu=off` on this host again; it silently kills the NPU.
|
||||
The same root cause explained the older
|
||||
`amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel line — a
|
||||
consequence of IOMMU-off, not a GPU fault.
|
||||
|
||||
## Client wiring
|
||||
**The standing rule this produced — "never set `iommu=off` on this host again" —
|
||||
has been superseded**: the NPU it protected is retired by decision, and
|
||||
`amd_iommu=off` is now intentional. See
|
||||
[[dankclaw-iommu-translation-off]] for the current posture and how to revert it.
|
||||
|
||||
The pi client model config (`/home/dank/.pi/agent/models.json` on Dankware)
|
||||
registers this endpoint under the Dankclaw provider with the **exact** FLM
|
||||
registry tag as the model `id` (unknown tags answer a 200-wrapped error, not
|
||||
a silent 1B fallback), `baseUrl` `http://192.168.0.143:7771/v1`, and a per-request context
|
||||
window equal to the serving ctx-len. Health checks for this service must use
|
||||
`GET /v1/version` (there is no `/health` route).
|
||||
## Client wiring (historical)
|
||||
|
||||
The pi/harness model config registered this endpoint with the **exact** FLM
|
||||
registry tag as the model `id` and `baseUrl` `http://192.168.0.143:7771/v1`.
|
||||
Both that provider block and Hermes's `dankclaw-npu` provider were removed at
|
||||
retirement; the routes are mapped in the table above.
|
||||
|
||||
## Citations
|
||||
|
||||
- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
|
||||
- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
|
||||
- [[halogen-flash-server]] — the surviving image-input path
|
||||
Reference in new issue
Block a user