serverAppConfigs/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	FastFlowLM NPU Inference	FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.	fastflowlm npu xdna2 dankclaw research	2026-09-03

SUPERSEDED 2026-08-17. The Frank redo retired this topology. See frank-redo-2026-08-17 and deployment for the active configuration. The frank-zealot name and port 52625 then belonged to the vLLM service on Dankclaw (nameistoken Quark W8A8); this page is archive.

The NPU endpoint is retired — 2026-09-11. On Dankclaw today frank-npu.service is masked/inactive and nothing listens on 52625 (ss -ltn, read 2026-10-18). The runtime is still installed (fastflowlm 1.0.1, libxrt2/libxrt-npu2, /opt/fastflowlm, ~/.config/flm, /usr/bin/flm — all read 2026-10-18) and the device is usable again since the 2026-10-18 iommu=pt boot (amdxdna bound, /dev/accel/accel0 present — dankclaw-iommu-translation-off), but no service runs on it. frank-zealot today is the llama.cpp PrismML endpoint on the RTX 3090 (frank-zealot-llama-prism-bonsai2-pq20). See frank-npu for the retirement record. Everything below is the 2026-08 research record.

FastFlowLM NPU Inference

FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and frank-zealot (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.

Verdict (2026-08-16)

• The NPU is a complement, not a replacement. It fits dense models up to about 9B.
• Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
• Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
• Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
• Landed 2026-08-16: gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
• The memlock limit was the setup blocker. Fixed 2026-08-16 via /etc/security/limits.d/99-flm-memlock.conf.

What FastFlowLM Is

FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.

Key properties, per the project README and docs:

• Runs fully on the NPU. No GPU or CPU load during inference.
• CLI mode (flm run <tag>) and server mode (flm serve <tag>).
• Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
• Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
• Context lengths up to 256k tokens on listed models.
• Vision, audio, embedding, and MoE model support.
• Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
• Models store in ~/.config/flm/ (Linux). Override with FLM_MODEL_PATH.

NPU Hardware on Dankclaw

Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", amd.com product page). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents 121 GiB today (free -g total 121, read 2026-10-18).

Live verification on Dankclaw, 2026-08-16:

$ flm validate
[FLM]  Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.0-29-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.65
[Linux]  amdxdna version: 0.7
[ERROR]  Memlock limit is too low (8MB). Please raise the limit or set to infinity.

• Kernel 7.0.0-29-generic shipped the amdxdna driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (lspci -k -s c6:00.1 → Kernel driver in use: amdxdna, read 2026-10-18).
• Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
• The XRT runtime packages (libxrt2, libxrt-npu2) are installed — still installed on 2026-10-18 (dpkg -l → libxrt2 1:2.21.75+dfsg-4, libxrt-npu2 1:2.21.75+dfsg-4).
• The only failure is the memlock ulimit (8 MB). FLM needs unlimited.
• Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".

Memory Model

The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.

Budget math for this fleet:

Machine	RAM	NPU budget (approx. half)
Dankclaw (Strix Halo)	30 GiB	~15 GiB
Dankserver (Ryzen 7 7700X)	62 GB	no NPU
FLM bench box (Kraken Point)	32 or 96 GB	~16 or ~48 GB

2026-10-18 re-read: Dankclaw's OS now reports 121 GiB (free -g) and Dankserver 246 GiB (free -g), so the half-of-RAM rule would price the NPU budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed.

Weight sizes (approximate, Q4-class):

Model	Quant	Weights	Fits ~15 GiB budget?
Qwen3.5-9B	Q4_1	~5.5 GB	yes
Qwen3-8B	Q4_1	~5 GB	yes
gpt-oss-20b (MoE, 3.6B active)	Q4_1	~15.1 GB	at the edge
Qwen3.6-35B-A3B (MoE, 3B active)	Q4_KS	~20 GB	no, unproven at 30 GiB
Qwen3.8-27B (dense)	any Q4	~16 GB+	no build exists

Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.

Supported Models (Qwen family, plus gpt-oss)

All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.

Tag	Base model	Context max	Notes
qwen3:0.6b	Qwen3-0.6B	32k	thinking toggle
qwen3:1.7b	Qwen3-1.7B	32k	thinking toggle
qwen3:4b	Qwen3-4B	32k	tool calling
qwen3:8b	Qwen3-8B	32k	tool calling
qwen3-tk:4b	Qwen3-4B-Thinking-2507	256k	thinking
qwen3-it:4b	Qwen3-4B-Instruct-2507	256k	tool calling
qwen3vl-it:4b	Qwen3-VL-4B-Instruct	256k	vision
qwen2.5-it:3b	Qwen2.5-3B-Instruct	32k
qwen2.5vl-it:3b	Qwen2.5-VL-3B-Instruct	256k	vision
qwen3.5:0.8b	Qwen3.5-0.8B	256k	vision, thinking toggle
qwen3.5:2b	Qwen3.5-2B	256k	vision, tool calling
qwen3.5:4b	Qwen3.5-4B	256k	vision, tool calling
qwen3.5:9b	Qwen3.5-9B	256k	vision, tool calling
qwen3.6-moe:35b-a3b	Qwen3.6-35B-A3B	256k	Q4_KS, MoE, vision, thinking, tool calling
gpt-oss:20b	gpt-oss-20b	128k	Q4_1, ~15.1 GB in FLM

Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.

Published Benchmarks (FLM, vendor numbers)

All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.

Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	39.2	38.0	36.3	33.1	28.1	21.6
Qwen3.5-2B	26.8	26.2	25.4	23.7	21.3	17.0
Qwen3.5-4B	15.0	14.6	14.2	13.3	11.8	9.6
Qwen3.5-9B	9.3	9.2	9.0	8.5	7.8	6.9

Prefill t/s for the same run:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	983	1257	1471	1584	1579	1447
Qwen3.5-2B	803	1004	1142	1223	1225	1151
Qwen3.5-4B	378	440	479	493	487	450
Qwen3.5-9B	284	333	362	379	378	357

Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point 96 GB DRAM:

Metric	1k	2k	4k	8k	16k	32k
Decode t/s	13.65	13.41	13.09	12.51	11.24	9.51
Prefill t/s	78.98	118.04	156.43	197.93	218.84	221.96

Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.

The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.

Fit to the Frank Program

Our model	NPU verdict
Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense)	No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget.
Qwen3.6-35B-A3B (Dankserver, MoE)	Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped.
Qwen3.5-9B / Qwen3-8B	Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS.
gpt-oss-20b (MoE)	Landed. Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results.

Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.

Install State on Dankclaw (verified 2026-08-16)

• fastflowlm 1.0.1 installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). flm at /usr/bin/flm. The 0.9.45 runtime refused the qwythos pack, which declares flm_version 1.0.0.
• libxrt2 / libxrt-npu2 (Xilinx Runtime) installed.
• /dev/accel/accel0 present. flm validate passed fully. (Re-read 2026-10-18: /dev/accel/accel0 is still present and fastflowlm 1.0.1 is still installed, but the endpoint is retired — frank-npu.service masked, no listener on 52625.)
• Memlock fixed: /etc/security/limits.d/99-flm-memlock.conf sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
• gpt-oss:20b pulled (4/4 files verified, 13.8 GB model.q4nx + tokenizer) into ~/.config/flm/models/GPT-OSS-20B-NPU2/.
• qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via flm-add.py (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry ~/.config/flm/model_list.json (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked ~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2. The service env exports FLM_CONFIG_PATH and FLM_XCLBIN_PATH so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
• flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
• The llama-server (frank-dragoon, the PP head) holds an fd on /dev/accel/accel0 from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.

Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)

Conditions: --pmode performance, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.

Run	Prompt tok	Compl tok	Wall s	Prefill t/s	Decode t/s	KV occ %
1 (smoke)	74	63	7.2	19.5	19.3	1.7
2 (160-tok prompt)	161	256	17.4	40.8	19.3	5.1
3 (95-tok prompt, 512 out)	95	512	30.8	24.3	19.3	n/a

Decode is flat at 19.3 t/s across all three runs. Output includes reasoning_content (gpt-oss is a reasoning model). At max_tokens 256 one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.

Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.

Model Bakeoff (measured 2026-08-16, 4 models)

All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, --pmode performance, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).

Model	Tag	Weights	Decode t/s (short)	8k prefill t/s	8k decode t/s	Vision decode t/s	Vision accuracy	Max ctx (registry)
Gemma4-E4B	gemma4-it:e4b	6.8 GB + 912 MB audio	12.2	582	9.9	11.7	4/4	32768
Qwen3.5-9B	qwen3.5:9b	7.3 GB	7.2	343	6.6	7.3	4/4	32768
GPT-OSS-20B	gpt-oss:20b	13.8 GB	19.3	378	16.9	n/a (no vision)	0/4	8192
Qwen3.6-35B-A3B	qwen3.6-moe:35b-a3b	22.2 GB	12.7	207	11.2	12.7	4/4	32768

Notes:

• gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
• 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
• Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
• All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.

Promotion (2026-08-16)

frank-zealot now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: scripts/frank-zealot.service (symlinked into /etc/systemd/system), wrapper scripts/serve-flm-small.sh, tunables scripts/frank-zealot.env. LimitMEMLOCK=infinity is set in the unit. Default model: gemma4-it:e4b. Swap FLM_MODEL in the env to switch.

Boot policy (2026-08-17)

The frank-zealot unit is disabled for boot. Only the PP pair (frank-dragoon + frank-pp-worker) starts at boot. The unit file was removed from /etc/systemd/system; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:

sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-zealot

Live config (2026-08-16): FLM_PMODE=turbo, FLM_ASR=1 (loads Whisper-V3-Turbo-NPU2; exposes /v1/audio/transcriptions), FLM_CTX_LEN=131072 (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).

The PP split was renamed frank-pp-head → frank-dragoon (2026-08-16, commit 48142f5).

Model swap (2026-08-16, commit 50c55ee): FLM_MODEL=qwen3.6-moe:35b-a3b, FLM_CTX_LEN=262144 (native max), FLM_PMODE=turbo, FLM_ASR=1 kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to gemma4-it:e4b. Swap FLM_MODEL and FLM_CTX_LEN back to restore it.

Model swap (2026-08-17, commit c85ff6c): FLM_MODEL=qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through flm-add (user registry ~/.config/flm/model_list.json + user xclbins ~/.config/flm/xclbins, kernels symlinked from official Qwen3.5-9B-NPU2). Runtime upgraded to FLM 1.0.1 first (see Install State). FLM_CTX_LEN=262144 native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). FLM_PMODE=turbo, FLM_ASR=1 kept. The env file now exports FLM_CONFIG_PATH / FLM_XCLBIN_PATH — required, or flm treats the tag as unknown and silently serves llama3.2:1b. qwen3.6-moe:35b-a3b files stay installed for rollback. pi client entry (~/.pi/agent/models.json on dankware): id qwythos:9b, ctx 262144, input [text] (pack has no vision weights), compat supportsDeveloperRole: false kept.

Parallel Streams (measured 2026-08-16)

The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags --socket (default 10) and --q-len (default 10) cap queue depth only. They do not add concurrency.

Concurrent-request test, gpt-oss-20b, 128-token cap per stream:

Streams	Wall s	Aggregate t/s	Per-stream wall s
1	10.6	12.1	10.6
2	21.1	12.0	10.8 / 21.1
4	41.1	11.6	10.9 / 21.6 / 32.4 / 41.1

The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.

Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.

Test Runbook

This runbook targets a retired endpoint; keep it as the procedure if the NPU is ever revived. Do not run large model loads while the production service holds most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading, 121 GiB today (free -g, read 2026-10-18). Check free RAM first, or stop the service the way the bench policy requires (sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon — self-healing would otherwise restart it). The NPU shares the unified pool with everything else on the box.

graph TD
    A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
    B --> C{NPU visible to XRT?}
    C -- no --> D[Fix driver or firmware path]
    C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
    E --> F{Generates tokens?}
    F -- no --> D
    F -- yes --> G[4. flm serve on a scratch port, never 7770]
    G --> H[5. Bench decode + prefill t/s]
    H --> I{Speed acceptable?}
    I -- no --> J[Record results, stop]
    I -- yes --> K[6. Attempt gpt-oss:20b]
    K --> L{Loads in ~15 GiB?}
    L -- no --> J
    L -- yes --> M[Bench, record]
    M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
    N --> O[Record, decision]

1. Raise memlock. Add * soft memlock unlimited and * hard memlock unlimited to /etc/security/limits.conf. Then reboot. For a systemd unit, set LimitMEMLOCK=infinity instead.
2. Verify the NPU. Run flm validate. Run xrt-smi examine. Both must show the NPU. flm validate uses the kernel device. flm run uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run flm run qwen3.5:4b. Toggle /verbose for performance reporting. Type /bye to exit.
4. Server mode. Run flm serve qwen3.5:4b. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt qwen3.6-moe:35b-a3b only after the smaller runs succeed, and only with the production service stopped. Watch free -h during the model download and load.

Registering packs absent from the catalog (NPU2)

Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model) to the FLM registry. Registry facts: user registry /home/dank/.config/flm/model_list.json, shape {"model_path": "models", "models": {"<tag>": {"<size>": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}; serve resolves tags through FLM_CONFIG_PATH at startup, and flm list with the env var set lists custom tags (⏬ = not downloaded).

1. Back up the registry JSON first (cp -a).
2. Insert the entry via a python json edit + atomic os.replace. Set files to the FULL HF file list — including vision_weight.q4nx even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log).
3. Do not use flm pull <tag> for custom entries — it throws json.exception out_of_range key '<tag>:<size>' not found while building the download list. Place files manually into /home/dank/.config/flm/models/<PackName>/ with curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true" — plain raw/main/<file> returns 135-byte LFS pointer stubs, not weights; verify byte counts against the HF tree API before trusting a download.
4. Kernel packs: copy from the same-shape family pack (shape-match config.json on hidden_size/num_hidden_layers/head_dim first) into /opt/fastflowlm/share/flm/xclbins/<PackName>/ via sudo cp -a (root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so the new process loads with all files present.
6. Verify with a REAL completion request (model: "<tag>:<size>") — /v1/models alone proves registration, not loadability.

Hard constraints: only ONE flm serve process can hold the NPU hardware context — a second concurrent serve process fails every load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 (aie2 mgmt msg opcode 0x2 status 0x2000003); never run scratch/verify instances while production holds the NPU. Tag switching happens INSIDE one serve process; the first request after a model switch pays ~6–11 s cold-load cost (visible in usage.load_duration). Over ssh, never pkill -f "<pattern>" with a pattern present in your own command line — use bracket-guarded patterns ([q]wen3.5:4b) or exact PIDs.

Tag load-failure diagnosis (200 with an error body)

POST /v1/chat/completions answers HTTP 200 with a flat {"error":"Failed to load <tag> model!"} and no choices; /v1/completions returns a nested OpenAI-shaped server_error (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the error key. Two distinct causes, distinguished via journalctl -u <flm-service> --full:

1. Unknown tag (not in the registry): immediate error, nothing loaded. Correct behavior, not a bug.
2. Registered tag whose kernel pack is missing: FLM resolves the tag, downloads missing weight files on the fly (first request hangs ~90–110 s), then fails Failed to load model: No such file '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'; subsequent attempts fail fast (~2 s). Root cause: the serve script exports FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins (user dir), so every kernel lookup resolves under the user dir, which only contains symlinks someone linked. A tag listed in /v1/models is NOT proof it is loadable on this host. Fix: one symlink per missing pack (ln -s /opt/fastflowlm/share/flm/xclbins/<PackName> /home/dank/.config/flm/xclbins/<PackName>) — verified live: after the symlink the first request loaded and answered in 7.4 s wall.

Switch-cost behavior: the model field routes per-request (no restart needed); a request whose model differs from the resident one triggers a full reload (7–11 s wall for an 8-token completion on ~4B-class packs); alternating tags pays the swap each time. Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (NPU Locked! → NPU Lock Released!). Probe recipe: /v1/models for registered tags; grep the journal for Failed to load model (the quoted xclbin path names the missing pack dir exactly); diff ls ~/.config/flm/xclbins/ (symlinks = installed) against /opt/fastflowlm/share/flm/xclbins/ (available).

Incident: qwen3.6-moe unreachable (2026-08-16)

Symptom: a client on dankware could not reach qwen3.6-moe:35b-a3b on port 52625. The journal shows three separate causes in the 14:19-15:03 window.

1. EXEC 203 crash loop (14:19-14:20). The exec bit was lost on serve-flm-small.sh on the target. The unit failed status=203/EXEC: Permission denied four times, then started after the bit was restored. The git mode is 100755 (commit 3f42210), and a git pull restores the bit. Verify with git ls-files -s scripts/serve-flm-small.sh.
2. Wrong model ID (14:29-14:52). The client config still used the model ID frank-zealot (a service alias). FLM rejects that tag with [ERROR] Model tag 'frank-zealot' is not supported. Each such request also triggered the unknown-tag fallback below.
3. Swap churn (14:48-15:03). The swap gemma4-it:e4b → qwen3.6-moe:35b-a3b restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.

Fix: the client model ID must be the FLM registry tag qwen3.6-moe:35b-a3b. The working pi agent entry in ~/.pi/agent/models.json is baseUrl http://192.168.0.143:52625/v1, id qwen3.6-moe:35b-a3b, contextWindow 262144, reasoning true, input [text, image]. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.

Unknown-tag fallback (FLM behavior)

When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model llama3.2:1b:

• If the llama3.2:1b files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
• FLM then loads the 1B model in place of the requested model.
• The files now exist in ~/.config/flm/models/Llama-3.2-1B-NPU2 (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.

Correction (live-probed 2026-08-30): the fallback no longer fires on the current serve path — an unknown tag now answers HTTP 200 with a flat {"error":"Failed to load <tag> model!"} body and nothing is loaded or served; /v1/completions answers a nested server_error (code 500) under HTTP 200.

Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move ~/.config/flm/models/Llama-3.2-1B-NPU2 (an incident artifact) to .AGENTTRASH so a stray tag re-downloads loudly instead of serving the wrong model.

Request compatibility: the developer role (found 2026-08-16)

pi sessions failed against this endpoint with Stream ended without finish_reason while curl worked. Root cause, verified by field bisection:

• The FLM Qwen chat template accepts only the roles system, user, and assistant. Any other role raises Unexpected message role. in the template.
• FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no finish_reason.
• pi sends role: "developer" when model.reasoning && compat.supportsDeveloperRole (pi source, openai-completions.js). The default supportsDeveloperRole is true for standard OpenAI-compatible providers, so a reasoning model without a compat block always triggers the bug.
• Fix: "compat": { "supportsDeveloperRole": false } on the model entry in ~/.pi/agent/models.json. pi then sends role: "system". Verified with a headless pi session: full round trip, finish_reason: stop.
• FLM ignores unknown body fields. reasoning_effort, store, max_completion_tokens, and tools: [] all arrive in pi requests and pass through without error.

Risks and Open Questions

• Ubuntu 26.04 is unofficial. The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
• The 30 GiB budget is untested at the very top. gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
• The FLM serve process was not under systemd (2026-08-era risk). A reboot of Dankclaw killed it. The fleet law today is the opposite: every deployed container/service ships a boot-persistent, self-healing unit, and the retired NPU unit is deliberately masked (frank-npu.service, read 2026-10-18) so it cannot come back at boot.
• gpt-oss reasoning budget. The model can spend its full token budget on reasoning_content and return empty content. Clients must handle or cap this.
• No llama.cpp GGUF path. FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
• NPU production role (2026-08-16, retired 2026-09-11). frank-zealot then served qwen3.6-moe:35b-a3b on port 52625; the endpoint was retired with the amd_iommu=off decision that day and the unit is masked today. The remaining NPU content (benchmarks, runbook steps) is research on a device that is again usable since 2026-10-18 but serves nothing.

Citations

1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM wikipedia_en_all_maxi_2026-02.
9. Live system checks on Dankclaw, 2026-08-16: flm validate output, dpkg -l package list, kernel 7.0.0-29-generic, /proc/3052096/fd device fd inspection.