serverAppConfigs/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	FastFlowLM NPU Inference	FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.	fastflowlm npu xdna2 dankclaw research	2026-09-03

SUPERSEDED 2026-08-17. The Frank redo retired this topology. See frank-redo-2026-08-17 and deployment for the active configuration. The frank-zealot name and port 52625 then belonged to the vLLM service on Dankclaw (nameistoken Quark W8A8); this page is archive.

FastFlowLM NPU Inference

FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and frank-zealot (port 52625) serves in production. See Promotion.

Verdict (2026-08-16)

• The NPU is a complement, not a replacement. It fits dense models up to about 9B.
• Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
• Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
• Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
• Landed 2026-08-16: gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
• The memlock limit was the setup blocker. Fixed 2026-08-16 via /etc/security/limits.d/99-flm-memlock.conf.

What FastFlowLM Is

FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.

Key properties, per the project README and docs:

• Runs fully on the NPU. No GPU or CPU load during inference.
• CLI mode (flm run <tag>) and server mode (flm serve <tag>).
• Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
• Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
• Context lengths up to 256k tokens on listed models.
• Vision, audio, embedding, and MoE model support.
• Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
• Models store in ~/.config/flm/ (Linux). Override with FLM_MODEL_PATH.

NPU Hardware on Dankclaw

Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool.

Live verification on Dankclaw, 2026-08-16:

$ flm validate
[FLM]  Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.0-29-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.65
[Linux]  amdxdna version: 0.7
[ERROR]  Memlock limit is too low (8MB). Please raise the limit or set to infinity.

• Kernel 7.0.0-29-generic ships the amdxdna driver in-tree. The driver is loaded.
• Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
• The XRT runtime packages (libxrt2, libxrt-npu2) are installed.
• The only failure is the memlock ulimit (8 MB). FLM needs unlimited.
• Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".

Memory Model

The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.

Budget math for this fleet:

Machine	RAM	NPU budget (approx. half)
Dankclaw (Strix Halo)	30 GiB	~15 GiB
Dankserver (Ryzen 7 7700X)	62 GB	no NPU
FLM bench box (Kraken Point)	32 or 96 GB	~16 or ~48 GB

Weight sizes (approximate, Q4-class):

Model	Quant	Weights	Fits ~15 GiB budget?
Qwen3.5-9B	Q4_1	~5.5 GB	yes
Qwen3-8B	Q4_1	~5 GB	yes
gpt-oss-20b (MoE, 3.6B active)	Q4_1	~15.1 GB	at the edge
Qwen3.6-35B-A3B (MoE, 3B active)	Q4_KS	~20 GB	no, unproven at 30 GiB
Qwen3.8-27B (dense)	any Q4	~16 GB+	no build exists

Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.

Supported Models (Qwen family, plus gpt-oss)

All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.

Tag	Base model	Context max	Notes
qwen3:0.6b	Qwen3-0.6B	32k	thinking toggle
qwen3:1.7b	Qwen3-1.7B	32k	thinking toggle
qwen3:4b	Qwen3-4B	32k	tool calling
qwen3:8b	Qwen3-8B	32k	tool calling
qwen3-tk:4b	Qwen3-4B-Thinking-2507	256k	thinking
qwen3-it:4b	Qwen3-4B-Instruct-2507	256k	tool calling
qwen3vl-it:4b	Qwen3-VL-4B-Instruct	256k	vision
qwen2.5-it:3b	Qwen2.5-3B-Instruct	32k
qwen2.5vl-it:3b	Qwen2.5-VL-3B-Instruct	256k	vision
qwen3.5:0.8b	Qwen3.5-0.8B	256k	vision, thinking toggle
qwen3.5:2b	Qwen3.5-2B	256k	vision, tool calling
qwen3.5:4b	Qwen3.5-4B	256k	vision, tool calling
qwen3.5:9b	Qwen3.5-9B	256k	vision, tool calling
qwen3.6-moe:35b-a3b	Qwen3.6-35B-A3B	256k	Q4_KS, MoE, vision, thinking, tool calling
gpt-oss:20b	gpt-oss-20b	128k	Q4_1, ~15.1 GB in FLM

Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.

Published Benchmarks (FLM, vendor numbers)

All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.

Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	39.2	38.0	36.3	33.1	28.1	21.6
Qwen3.5-2B	26.8	26.2	25.4	23.7	21.3	17.0
Qwen3.5-4B	15.0	14.6	14.2	13.3	11.8	9.6
Qwen3.5-9B	9.3	9.2	9.0	8.5	7.8	6.9

Prefill t/s for the same run:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	983	1257	1471	1584	1579	1447
Qwen3.5-2B	803	1004	1142	1223	1225	1151
Qwen3.5-4B	378	440	479	493	487	450
Qwen3.5-9B	284	333	362	379	378	357

Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point 96 GB DRAM:

Metric	1k	2k	4k	8k	16k	32k
Decode t/s	13.65	13.41	13.09	12.51	11.24	9.51
Prefill t/s	78.98	118.04	156.43	197.93	218.84	221.96

Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.

The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.

Fit to the Frank Program

Our model	NPU verdict
Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense)	No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget.
Qwen3.6-35B-A3B (Dankserver, MoE)	Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped.
Qwen3.5-9B / Qwen3-8B	Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS.
gpt-oss-20b (MoE)	Landed. Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results.

Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.

Install State on Dankclaw (verified 2026-08-16)

• fastflowlm 1.0.1 installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). flm at /usr/bin/flm. The 0.9.45 runtime refused the qwythos pack, which declares flm_version 1.0.0.
• libxrt2 / libxrt-npu2 (Xilinx Runtime) installed.
• /dev/accel/accel0 present. flm validate passes fully.
• Memlock fixed: /etc/security/limits.d/99-flm-memlock.conf sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
• gpt-oss:20b pulled (4/4 files verified, 13.8 GB model.q4nx + tokenizer) into ~/.config/flm/models/GPT-OSS-20B-NPU2/.
• qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via flm-add.py (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry ~/.config/flm/model_list.json (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked ~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2. The service env exports FLM_CONFIG_PATH and FLM_XCLBIN_PATH so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
• flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
• The llama-server (frank-dragoon, the PP head) holds an fd on /dev/accel/accel0 from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.

Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)

Conditions: --pmode performance, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.

Run	Prompt tok	Compl tok	Wall s	Prefill t/s	Decode t/s	KV occ %
1 (smoke)	74	63	7.2	19.5	19.3	1.7
2 (160-tok prompt)	161	256	17.4	40.8	19.3	5.1
3 (95-tok prompt, 512 out)	95	512	30.8	24.3	19.3	n/a

Decode is flat at 19.3 t/s across all three runs. Output includes reasoning_content (gpt-oss is a reasoning model). At max_tokens 256 one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.

Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.

Model Bakeoff (measured 2026-08-16, 4 models)

All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, --pmode performance, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).

Model	Tag	Weights	Decode t/s (short)	8k prefill t/s	8k decode t/s	Vision decode t/s	Vision accuracy	Max ctx (registry)
Gemma4-E4B	gemma4-it:e4b	6.8 GB + 912 MB audio	12.2	582	9.9	11.7	4/4	32768
Qwen3.5-9B	qwen3.5:9b	7.3 GB	7.2	343	6.6	7.3	4/4	32768
GPT-OSS-20B	gpt-oss:20b	13.8 GB	19.3	378	16.9	n/a (no vision)	0/4	8192
Qwen3.6-35B-A3B	qwen3.6-moe:35b-a3b	22.2 GB	12.7	207	11.2	12.7	4/4	32768

Notes:

• gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
• 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
• Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
• All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.

Promotion (2026-08-16)

frank-zealot now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: scripts/frank-zealot.service (symlinked into /etc/systemd/system), wrapper scripts/serve-flm-small.sh, tunables scripts/frank-zealot.env. LimitMEMLOCK=infinity is set in the unit. Default model: gemma4-it:e4b. Swap FLM_MODEL in the env to switch.

Boot policy (2026-08-17)

The frank-zealot unit is disabled for boot. Only the PP pair (frank-dragoon + frank-pp-worker) starts at boot. The unit file was removed from /etc/systemd/system; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:

sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-zealot

Live config (2026-08-16): FLM_PMODE=turbo, FLM_ASR=1 (loads Whisper-V3-Turbo-NPU2; exposes /v1/audio/transcriptions), FLM_CTX_LEN=131072 (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).

The PP split was renamed frank-pp-head → frank-dragoon (2026-08-16, commit 48142f5).

Model swap (2026-08-16, commit 50c55ee): FLM_MODEL=qwen3.6-moe:35b-a3b, FLM_CTX_LEN=262144 (native max), FLM_PMODE=turbo, FLM_ASR=1 kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to gemma4-it:e4b. Swap FLM_MODEL and FLM_CTX_LEN back to restore it.

Model swap (2026-08-17, commit c85ff6c): FLM_MODEL=qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through flm-add (user registry ~/.config/flm/model_list.json + user xclbins ~/.config/flm/xclbins, kernels symlinked from official Qwen3.5-9B-NPU2). Runtime upgraded to FLM 1.0.1 first (see Install State). FLM_CTX_LEN=262144 native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). FLM_PMODE=turbo, FLM_ASR=1 kept. The env file now exports FLM_CONFIG_PATH / FLM_XCLBIN_PATH — required, or flm treats the tag as unknown and silently serves llama3.2:1b. qwen3.6-moe:35b-a3b files stay installed for rollback. pi client entry (~/.pi/agent/models.json on dankware): id qwythos:9b, ctx 262144, input [text] (pack has no vision weights), compat supportsDeveloperRole: false kept.

Parallel Streams (measured 2026-08-16)

The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags --socket (default 10) and --q-len (default 10) cap queue depth only. They do not add concurrency.

Concurrent-request test, gpt-oss-20b, 128-token cap per stream:

Streams	Wall s	Aggregate t/s	Per-stream wall s
1	10.6	12.1	10.6
2	21.1	12.0	10.8 / 21.1
4	41.1	11.6	10.9 / 21.6 / 32.4 / 41.1

The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.

Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.

Test Runbook

Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.

graph TD
    A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
    B --> C{NPU visible to XRT?}
    C -- no --> D[Fix driver or firmware path]
    C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
    E --> F{Generates tokens?}
    F -- no --> D
    F -- yes --> G[4. flm serve on a scratch port, never 7770]
    G --> H[5. Bench decode + prefill t/s]
    H --> I{Speed acceptable?}
    I -- no --> J[Record results, stop]
    I -- yes --> K[6. Attempt gpt-oss:20b]
    K --> L{Loads in ~15 GiB?}
    L -- no --> J
    L -- yes --> M[Bench, record]
    M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
    N --> O[Record, decision]

1. Raise memlock. Add * soft memlock unlimited and * hard memlock unlimited to /etc/security/limits.conf. Then reboot. For a systemd unit, set LimitMEMLOCK=infinity instead.
2. Verify the NPU. Run flm validate. Run xrt-smi examine. Both must show the NPU. flm validate uses the kernel device. flm run uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run flm run qwen3.5:4b. Toggle /verbose for performance reporting. Type /bye to exit.
4. Server mode. Run flm serve qwen3.5:4b. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt qwen3.6-moe:35b-a3b only after the smaller runs succeed, and only with the production service stopped. Watch free -h during the model download and load.

Registering packs absent from the catalog (NPU2)

Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model) to the FLM registry. Registry facts: user registry /home/dank/.config/flm/model_list.json, shape {"model_path": "models", "models": {"<tag>": {"<size>": {name, url, file_url, size, flm_min_version, files[], vlm, default_context_length, max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}; serve resolves tags through FLM_CONFIG_PATH at startup, and flm list with the env var set lists custom tags (⏬ = not downloaded).

1. Back up the registry JSON first (cp -a).
2. Insert the entry via a python json edit + atomic os.replace. Set files to the FULL HF file list — including vision_weight.q4nx even for text-only packs: the qwen3.5-family loader opens it unconditionally and a missing file makes the first request HANG silently (no error log).
3. Do not use flm pull <tag> for custom entries — it throws json.exception out_of_range key '<tag>:<size>' not found while building the download list. Place files manually into /home/dank/.config/flm/models/<PackName>/ with curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true" — plain raw/main/<file> returns 135-byte LFS pointer stubs, not weights; verify byte counts against the HF tree API before trusting a download.
4. Kernel packs: copy from the same-shape family pack (shape-match config.json on hidden_size/num_hidden_layers/head_dim first) into /opt/fastflowlm/share/flm/xclbins/<PackName>/ via sudo cp -a (root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so the new process loads with all files present.
6. Verify with a REAL completion request (model: "<tag>:<size>") — /v1/models alone proves registration, not loadability.

Hard constraints: only ONE flm serve process can hold the NPU hardware context — a second concurrent serve process fails every load with DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22 (aie2 mgmt msg opcode 0x2 status 0x2000003); never run scratch/verify instances while production holds the NPU. Tag switching happens INSIDE one serve process; the first request after a model switch pays ~6–11 s cold-load cost (visible in usage.load_duration). Over ssh, never pkill -f "<pattern>" with a pattern present in your own command line — use bracket-guarded patterns ([q]wen3.5:4b) or exact PIDs.

Tag load-failure diagnosis (200 with an error body)

POST /v1/chat/completions answers HTTP 200 with a flat {"error":"Failed to load <tag> model!"} and no choices; /v1/completions returns a nested OpenAI-shaped server_error (code 500) also under HTTP 200. A status-code-only client reads this as a successful empty completion — always check for the error key. Two distinct causes, distinguished via journalctl -u <flm-service> --full:

1. Unknown tag (not in the registry): immediate error, nothing loaded. Correct behavior, not a bug.
2. Registered tag whose kernel pack is missing: FLM resolves the tag, downloads missing weight files on the fly (first request hangs ~90–110 s), then fails Failed to load model: No such file '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'; subsequent attempts fail fast (~2 s). Root cause: the serve script exports FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins (user dir), so every kernel lookup resolves under the user dir, which only contains symlinks someone linked. A tag listed in /v1/models is NOT proof it is loadable on this host. Fix: one symlink per missing pack (ln -s /opt/fastflowlm/share/flm/xclbins/<PackName> /home/dank/.config/flm/xclbins/<PackName>) — verified live: after the symlink the first request loaded and answered in 7.4 s wall.

Switch-cost behavior: the model field routes per-request (no restart needed); a request whose model differs from the resident one triggers a full reload (7–11 s wall for an 8-token completion on ~4B-class packs); alternating tags pays the swap each time. Single-lane NPU: concurrent requests queue FIFO behind the current lock holder (NPU Locked! → NPU Lock Released!). Probe recipe: /v1/models for registered tags; grep the journal for Failed to load model (the quoted xclbin path names the missing pack dir exactly); diff ls ~/.config/flm/xclbins/ (symlinks = installed) against /opt/fastflowlm/share/flm/xclbins/ (available).

Incident: qwen3.6-moe unreachable (2026-08-16)

Symptom: a client on dankware could not reach qwen3.6-moe:35b-a3b on port 52625. The journal shows three separate causes in the 14:19-15:03 window.

1. EXEC 203 crash loop (14:19-14:20). The exec bit was lost on serve-flm-small.sh on the target. The unit failed status=203/EXEC: Permission denied four times, then started after the bit was restored. The git mode is 100755 (commit 3f42210), and a git pull restores the bit. Verify with git ls-files -s scripts/serve-flm-small.sh.
2. Wrong model ID (14:29-14:52). The client config still used the model ID frank-zealot (a service alias). FLM rejects that tag with [ERROR] Model tag 'frank-zealot' is not supported. Each such request also triggered the unknown-tag fallback below.
3. Swap churn (14:48-15:03). The swap gemma4-it:e4b → qwen3.6-moe:35b-a3b restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.

Fix: the client model ID must be the FLM registry tag qwen3.6-moe:35b-a3b. The working pi agent entry in ~/.pi/agent/models.json is baseUrl http://192.168.0.143:52625/v1, id qwen3.6-moe:35b-a3b, contextWindow 262144, reasoning true, input [text, image]. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.

Unknown-tag fallback (FLM behavior)

When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model llama3.2:1b:

• If the llama3.2:1b files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
• FLM then loads the 1B model in place of the requested model.
• The files now exist in ~/.config/flm/models/Llama-3.2-1B-NPU2 (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.

Correction (live-probed 2026-08-30): the fallback no longer fires on the current serve path — an unknown tag now answers HTTP 200 with a flat {"error":"Failed to load <tag> model!"} body and nothing is loaded or served; /v1/completions answers a nested server_error (code 500) under HTTP 200.

Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move ~/.config/flm/models/Llama-3.2-1B-NPU2 (an incident artifact) to .AGENTTRASH so a stray tag re-downloads loudly instead of serving the wrong model.

Request compatibility: the developer role (found 2026-08-16)

pi sessions failed against this endpoint with Stream ended without finish_reason while curl worked. Root cause, verified by field bisection:

• The FLM Qwen chat template accepts only the roles system, user, and assistant. Any other role raises Unexpected message role. in the template.
• FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no finish_reason.
• pi sends role: "developer" when model.reasoning && compat.supportsDeveloperRole (pi source, openai-completions.js). The default supportsDeveloperRole is true for standard OpenAI-compatible providers, so a reasoning model without a compat block always triggers the bug.
• Fix: "compat": { "supportsDeveloperRole": false } on the model entry in ~/.pi/agent/models.json. pi then sends role: "system". Verified with a headless pi session: full round trip, finish_reason: stop.
• FLM ignores unknown body fields. reasoning_effort, store, max_completion_tokens, and tools: [] all arrive in pi requests and pass through without error.

Risks and Open Questions

• Ubuntu 26.04 is unofficial. The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
• The 30 GiB budget is untested at the very top. gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
• The FLM serve process is not under systemd. A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
• gpt-oss reasoning budget. The model can spend its full token budget on reasoning_content and return empty content. Clients must handle or cap this.
• No llama.cpp GGUF path. FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
• NPU production role (2026-08-16). frank-zealot serves qwen3.6-moe:35b-a3b on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.

Citations

1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM wikipedia_en_all_maxi_2026-02.
9. Live system checks on Dankclaw, 2026-08-16: flm validate output, dpkg -l package list, kernel 7.0.0-29-generic, /proc/3052096/fd device fd inspection.