serverAppConfigs/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

---
type: reference
title: FastFlowLM NPU Inference
description: "The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook."
tags:
  - fastflowlm
  - npu
  - xdna2
  - dankclaw
  - research
timestamp: 2026-09-03
---

> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
>
> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
> (`ss -ltn`, read 2026-10-18). The runtime is still installed
> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.

# FastFlowLM NPU Inference

FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.

## Verdict (2026-08-16)

- The NPU is a complement, not a replacement. It fits dense models up to about 9B.
- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.

## What FastFlowLM Is

FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.

Key properties, per the project README and docs:

- Runs fully on the NPU. No GPU or CPU load during inference.
- CLI mode (`flm run <tag>`) and server mode (`flm serve <tag>`).
- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
- Context lengths up to 256k tokens on listed models.
- Vision, audio, embedding, and MoE model support.
- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`.

## NPU Hardware on Dankclaw

Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU
with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory
via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at
4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]].

Live verification on Dankclaw, 2026-08-16:

```
$ flm validate
[FLM]  Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.0-29-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.65
[Linux]  amdxdna version: 0.7
[ERROR]  Memlock limit is too low (8MB). Please raise the limit or set to infinity.
```

- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".

## Memory Model

The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.

Budget math for this fleet:

| Machine | RAM | NPU budget (approx. half) |
|---------|-----|---------------------------|
| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB |
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |

The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]]
and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate`
(it owns the current number).

Weight sizes (approximate, Q4-class):

| Model | Quant | Weights | Fits ~15 GiB budget? |
|-------|-------|---------|----------------------|
| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes |
| Qwen3-8B | Q4_1 | ~5 GB | yes |
| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge |
| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB |
| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists |

Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.

## Supported Models (Qwen family, plus gpt-oss)

All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.

| Tag | Base model | Context max | Notes |
|-----|-----------|-------------|-------|
| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle |
| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle |
| `qwen3:4b` | Qwen3-4B | 32k | tool calling |
| `qwen3:8b` | Qwen3-8B | 32k | tool calling |
| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking |
| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling |
| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision |
| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | |
| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision |
| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle |
| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling |
| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling |
| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling |
| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling |
| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM |

Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.

## Published Benchmarks (FLM, vendor numbers)

All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.

Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:

| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 |
| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 |
| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 |
| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 |

Prefill t/s for the same run:

| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 |
| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 |
| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 |
| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 |

Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:

| Metric | 1k | 2k | 4k | 8k | 16k | 32k |
|--------|-----:|-----:|-----:|-----:|-----:|-----:|
| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |

- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current
  PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at
  14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade
  the worker's model.

The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.

## Fit to the Frank Program

| Our model | NPU verdict |
|-----------|-------------|
| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |

Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.

## Install State on Dankclaw (verified 2026-08-16)

- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.

## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)

Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.

| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |

Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.

Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.

## Model Bakeoff (measured 2026-08-16, 4 models)

All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).

| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |

Notes:

- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.

## Promotion (2026-08-16)

`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused
twice after this window - see the banner - and the NPU unit today is `frank-npu`,
masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`),
wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`.
`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap
`FLM_MODEL` in the env to switch.

### Boot policy (2026-08-17)

The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:

```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-zealot
```

Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).

The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).

Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.

Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.

## Parallel Streams (measured 2026-08-16)

The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.

Concurrent-request test, gpt-oss-20b, 128-token cap per stream:

| Streams | Wall s | Aggregate t/s | Per-stream wall s |
|--------:|-------:|--------------:|-------------------|
| 1 | 10.6 | 12.1 | 10.6 |
| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |

The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.

Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.

## Test Runbook
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived.

**Host RAM availability.** The gate is half of system RAM by default - a relic of the
install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's
box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM
first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
the box.

```mermaid
graph TD
    A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
    B --> C{NPU visible to XRT?}
    C -- no --> D[Fix driver or firmware path]
    C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
    E --> F{Generates tokens?}
    F -- no --> D
    F -- yes --> G[4. flm serve on a scratch port, never 7770]
    G --> H[5. Bench decode + prefill t/s]
    H --> I{Speed acceptable?}
    I -- no --> J[Record results, stop]
    I -- yes --> K[6. Attempt gpt-oss:20b]
    K --> L{Loads in ~15 GiB?}
    L -- no --> J
    L -- yes --> M[Bench, record]
    M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
    N --> O[Record, decision]
```

1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead.
2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit.
4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.

## Registering packs absent from the catalog (NPU2)

Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
to the FLM registry. Registry facts: user registry
`/home/dank/.config/flm/model_list.json`, shape
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
file_url, size, flm_min_version, files[], vlm, default_context_length,
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
env var set lists custom tags (`⏬` = not downloaded).

1. Back up the registry JSON first (`cp -a`).
2. Insert the entry via a python json edit + atomic `os.replace`. Set
   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
   for text-only packs**: the qwen3.5-family loader opens it unconditionally
   and a missing file makes the first request HANG silently (no error log).
3. **Do not use `flm pull <tag>` for custom entries** — it throws
   `json.exception out_of_range key '<tag>:<size>' not found` while building
   the download list. Place files manually into
   `/home/dank/.config/flm/models/<PackName>/` with
   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
   weights; verify byte counts against the HF tree API before trusting a
   download.
4. Kernel packs: copy from the same-shape family pack (shape-match
   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
   (root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
   the new process loads with all files present.
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
   `/v1/models` alone proves registration, not loadability.

Hard constraints: only ONE `flm serve` process can hold the NPU hardware
context — a second concurrent serve process fails every load with
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
0x2000003); never run scratch/verify instances while production holds the
NPU. Tag switching happens INSIDE one serve process; the first request
after a model switch pays ~6–11 s cold-load cost (visible in
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
pattern present in your own command line — use bracket-guarded patterns
(`[q]wen3.5:4b`) or exact PIDs.

## Tag load-failure diagnosis (200 with an error body)

`POST /v1/chat/completions` answers **HTTP 200** with a flat
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
200. A status-code-only client reads this as a successful empty completion —
**always check for the `error` key**. Two distinct causes, distinguished via
`journalctl -u <flm-service> --full`:

1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
   Correct behavior, not a bug.
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
   downloads missing weight files on the fly (first request hangs ~90–110 s),
   then fails `Failed to load model: No such file
   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
   attempts fail fast (~2 s). Root cause: the serve script exports
   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
   kernel lookup resolves under the user dir, which only contains symlinks
   someone linked. **A tag listed in `/v1/models` is NOT proof it is
   loadable on this host.** Fix: one symlink per missing pack
   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
   symlink the first request loaded and answered in 7.4 s wall.

Switch-cost behavior: the `model` field routes per-request (no restart
needed); a request whose model differs from the resident one triggers a full
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
alternating tags pays the swap each time. Single-lane NPU: concurrent
requests queue FIFO behind the current lock holder (`NPU Locked!` →
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
the journal for `Failed to load model` (the quoted xclbin path names the
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).

## Incident: qwen3.6-moe unreachable (2026-08-16)

Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.

1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.

Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.

### Unknown-tag fallback (FLM behavior)

When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`:

- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
- FLM then loads the 1B model in place of the requested model.
- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.

**Correction (live-probed 2026-08-30):** the fallback no longer fires on the
current serve path — an unknown tag now answers HTTP 200 with a flat
`{"error":"Failed to load <tag> model!"}` body and nothing is loaded or served;
`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200.

Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model.

### Request compatibility: the `developer` role (found 2026-08-16)

pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection:

- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template.
- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`.
- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug.
- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`.
- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error.

## Risks and Open Questions

- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
  of Dankclaw killed it. The fleet law today is the opposite: every deployed
  container/service ships a boot-persistent, self-healing unit, and the retired
  NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
  cannot come back at boot.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
  served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
  `amd_iommu=off` decision that day and the unit is masked today. The remaining
  NPU content (benchmarks, runbook steps) is research on a device that is again
  usable since 2026-10-18 but serves nothing.

## Citations

1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`.
9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection.