Files

T

Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu

frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.

2026-09-11 14:48:28 -07:00

6.6 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-npu — the retired FLM NPU endpoint

RETIRED 2026-09-11. The endpoint is gone by operator decision, together with the IOMMU configuration it required — see dankclaw-iommu-translation-off. The service unit, env file and serve script are staged in serverAppConfigs/.AGENTTRASH/ (suffixed .flm-npu), and the host unit is stopped and masked.

The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server (/usr/bin/flm, dpkg fastflowlm) on the XDNA2 NPU (50 TOPS), serving the whole local model registry on port 7771 and routing per request by the exact FLM registry tag. Its one capability the iGPU fleet lacked was sight — and briefly sound: the Gemma4 E4B IT NPU2 pack carries both a vision and an audio tower, which is why the service ran that pack rather than the uncensored-but-text-only alternative.

This page is kept as the retirement record. The operational detail below was live-verified while the service ran and remains accurate about FLM's shape.

Why it is retired

The replacement large-model runtime is worth 13–16 % of prefill if the kernel boots with amd_iommu=off instead of iommu=pt, measured by the upstream project on this exact silicon. That single flag is the whole decision:

With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand out /dev/accel/accel0 — every open fails with err=-19 ("No such device"). That is precisely the failure the endpoint was rescued from on 2026-08-30, recorded in the next section.
iommu=pt is not a middle ground for the NPU and the prefill number: the passthrough domain still costs the prefill throughput the vendor measured.
The operator chose prefill, so the NPU endpoint goes.

Consequences accepted at retirement: audio input left the fleet with no replacement (halogen reads images only, no audio or video), and the surviving image path is halogen's vision tower on frank-large.

Where its routes went

consumer	setting	now
Hermes	`auxiliary.vision`	`dankclaw-large` / `frank-large` (halogen vision tower)
Hermes	`auxiliary.title_generation`	`dankserver-small` / `frank-small`
harness (`~/.omp/agent/models.yml`)	`modelRoles.vision`	`llamacpp (dankclaw)` / `frank-large`
harness	`modelRoles.tiny`	`llamacpp (dankserver)` / `frank-small`
harness	`modelRoles.task`	`llamacpp (dankclaw)` / `frank-large:medium`

The flm (dankclaw-npu) provider block was removed from the harness model config, and the dankclaw-npu provider from Hermes's live config.yaml.

What it was

Field	Value
Runtime	FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU
Service	`frank-npu.service` (systemd system unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`)
Files (archived)	`scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service`
Port	`7771` on `0.0.0.0`
Power mode	`FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything
Context	`FLM_CTX_LEN=131072` (128K, the verified maximum on this box)
Vision/audio	`FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions`

The serve script exported FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json and FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins, with kernels symlinked from /opt/fastflowlm/share/flm/xclbins/ into the per-pack directories, so a tag swap was an env-file edit rather than a script edit.

API contract (as it behaved)

The endpoint served the whole local registry (36 tags at first deploy) on one port; /v1/models listed them all. Chat completions routed per request by the exact registry tag in the model field — never a service alias.
Unknown tags did not 404: /v1/chat/completions answered HTTP 200 with a flat error body {"error":"Failed to load <tag> model!"} and no choices. Status-code-only clients silently got an empty response.
Health was GET /v1/version; FLM has no /health route.
The NPU is a single-lane device — generations serialized FIFO. It was a background/triage endpoint, not interactive multi-user chat.

The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)

First deploy crash-looped every 10 s with Error: Open /dev/accel/accel0 failed (err=-19): No such device. The chain, all live-verified on the host:

amdxdna_drm_open failed SVA bind with ret -19 for every client — the XDNA2 driver cannot hand out the accel device without PASID/SVA.
SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then carried iommu=off. With IOMMU off there are no IOMMU groups and no PASID state; PASIDCtl: Enable- was observed on 0000:c5:00.1.
The fix was iommu=pt: passthrough keeps the iGPU default domain identity-mapped (no DMA-translation overhead for frank-large) while enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
Post-boot verification: /sys/kernel/iommu_groups/ populated (30 groups), zero SVA bind device failed entries since boot, accel0 recreated at boot, service active with NRestarts=0.

The same root cause explained the older amdgpu 0000:c4:00.0: no vmid pasid mapping supported kernel line — a consequence of IOMMU-off, not a GPU fault.

The standing rule this produced — "never set iommu=off on this host again" — has been superseded: the NPU it protected is retired by decision, and amd_iommu=off is now intentional. See dankclaw-iommu-translation-off for the current posture and how to revert it.

Client wiring (historical)

The pi/harness model config registered this endpoint with the exact FLM registry tag as the model id and baseUrl http://192.168.0.143:7771/v1. Both that provider block and Hermes's dankclaw-npu provider were removed at retirement; the routes are mapped in the table above.

Citations

FastFlowLM — FLM runtime and NPU model packs
dankclaw-iommu-translation-off — the decision that retired this endpoint
halogen-flash-server — the surviving image-input path

6.6 KiB Raw Blame History Unescape Escape