serverAppConfigs/docs/FastFlowLM NPU Inference.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	FastFlowLM NPU Inference	FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.	fastflowlm npu xdna2 dankclaw research	2026-08-16

FastFlowLM NPU Inference

FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw and the NPU validates, but nothing serves from it yet.

Verdict (2026-08-16)

• The NPU is a complement, not a replacement. It fits dense models up to about 9B.
• Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
• Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
• Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
• One setup blocker remains: the memlock limit. Fix it before any test.

What FastFlowLM Is

FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.

Key properties, per the project README and docs:

• Runs fully on the NPU. No GPU or CPU load during inference.
• CLI mode (flm run <tag>) and server mode (flm serve <tag>).
• Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
• Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
• Context lengths up to 256k tokens on listed models.
• Vision, audio, embedding, and MoE model support.
• Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
• Models store in ~/.config/flm/ (Linux). Override with FLM_MODEL_PATH.

NPU Hardware on Dankclaw

Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool.

Live verification on Dankclaw, 2026-08-16:

$ flm validate
[FLM]  Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.0-29-generic
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.65
[Linux]  amdxdna version: 0.7
[ERROR]  Memlock limit is too low (8MB). Please raise the limit or set to infinity.

• Kernel 7.0.0-29-generic ships the amdxdna driver in-tree. The driver is loaded.
• Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
• The XRT runtime packages (libxrt2, libxrt-npu2) are installed.
• The only failure is the memlock ulimit (8 MB). FLM needs unlimited.
• Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".

Memory Model

The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.

Budget math for this fleet:

Machine	RAM	NPU budget (approx. half)
Dankclaw (Strix Halo)	30 GiB	~15 GiB
Dankserver (Ryzen 7 7700X)	62 GB	no NPU
FLM bench box (Kraken Point)	32 or 96 GB	~16 or ~48 GB

Weight sizes (approximate, Q4-class):

Model	Quant	Weights	Fits ~15 GiB budget?
Qwen3.5-9B	Q4_1	~5.5 GB	yes
Qwen3-8B	Q4_1	~5 GB	yes
gpt-oss-20b (MoE, 3.6B active)	Q4_1	~15.1 GB	at the edge
Qwen3.6-35B-A3B (MoE, 3B active)	Q4_KS	~20 GB	no, unproven at 30 GiB
Qwen3.8-27B (dense)	any Q4	~16 GB+	no build exists

Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.

Supported Models (Qwen family, plus gpt-oss)

All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.

Tag	Base model	Context max	Notes
qwen3:0.6b	Qwen3-0.6B	32k	thinking toggle
qwen3:1.7b	Qwen3-1.7B	32k	thinking toggle
qwen3:4b	Qwen3-4B	32k	tool calling
qwen3:8b	Qwen3-8B	32k	tool calling
qwen3-tk:4b	Qwen3-4B-Thinking-2507	256k	thinking
qwen3-it:4b	Qwen3-4B-Instruct-2507	256k	tool calling
qwen3vl-it:4b	Qwen3-VL-4B-Instruct	256k	vision
qwen2.5-it:3b	Qwen2.5-3B-Instruct	32k
qwen2.5vl-it:3b	Qwen2.5-VL-3B-Instruct	256k	vision
qwen3.5:0.8b	Qwen3.5-0.8B	256k	vision, thinking toggle
qwen3.5:2b	Qwen3.5-2B	256k	vision, tool calling
qwen3.5:4b	Qwen3.5-4B	256k	vision, tool calling
qwen3.5:9b	Qwen3.5-9B	256k	vision, tool calling
qwen3.6-moe:35b-a3b	Qwen3.6-35B-A3B	256k	Q4_KS, MoE, vision, thinking, tool calling
gpt-oss:20b	gpt-oss-20b	128k	Q4_1, ~15.1 GB in FLM

Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.

Published Benchmarks (FLM, vendor numbers)

All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.

Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	39.2	38.0	36.3	33.1	28.1	21.6
Qwen3.5-2B	26.8	26.2	25.4	23.7	21.3	17.0
Qwen3.5-4B	15.0	14.6	14.2	13.3	11.8	9.6
Qwen3.5-9B	9.3	9.2	9.0	8.5	7.8	6.9

Prefill t/s for the same run:

Model	1k	2k	4k	8k	16k	32k
Qwen3.5-0.8B	983	1257	1471	1584	1579	1447
Qwen3.5-2B	803	1004	1142	1223	1225	1151
Qwen3.5-4B	378	440	479	493	487	450
Qwen3.5-9B	284	333	362	379	378	357

Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point 96 GB DRAM:

Metric	1k	2k	4k	8k	16k	32k
Decode t/s	13.65	13.41	13.09	12.51	11.24	9.51
Prefill t/s	78.98	118.04	156.43	197.93	218.84	221.96

Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.

The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.

Fit to the Frank Program

Our model	NPU verdict
Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense)	No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget.
Qwen3.6-35B-A3B (Dankserver, MoE)	Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped.
Qwen3.5-9B / Qwen3-8B	Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS.
gpt-oss-20b (MoE)	At the edge. 15.1 GB against a ~15 GiB cap. Vendor notes "32 GB machines may work inconsistently".

Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.

Install State on Dankclaw (verified 2026-08-16)

• fastflowlm 0.9.45-12-ga6bd056 installed (deb). flm at /usr/bin/flm.
• libxrt2 / libxrt-npu2 (Xilinx Runtime) installed.
• /dev/accel/accel0 present. flm validate passes except memlock.
• No models downloaded yet (~/.config/flm/ empty).
• No FLM process running.
• The llama-server (frank-pp-head) holds fd 4 open on /dev/accel/accel0 since its own start at 10:43. This is device enumeration at startup. It does not compute on the NPU. Verify FLM can open the device while llama-server runs.

Test Runbook

Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.

graph TD
    A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
    B --> C{NPU visible to XRT?}
    C -- no --> D[Fix driver or firmware path]
    C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
    E --> F{Generates tokens?}
    F -- no --> D
    F -- yes --> G[4. flm serve on a scratch port, never 7770]
    G --> H[5. Bench decode + prefill t/s]
    H --> I{Speed acceptable?}
    I -- no --> J[Record results, stop]
    I -- yes --> K[6. Attempt gpt-oss:20b]
    K --> L{Loads in ~15 GiB?}
    L -- no --> J
    L -- yes --> M[Bench, record]
    M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
    N --> O[Record, decision]

1. Raise memlock. Add * soft memlock unlimited and * hard memlock unlimited to /etc/security/limits.conf. Then reboot. For a systemd unit, set LimitMEMLOCK=infinity instead.
2. Verify the NPU. Run flm validate. Run xrt-smi examine. Both must show the NPU. flm validate uses the kernel device. flm run uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run flm run qwen3.5:4b. Toggle /verbose for performance reporting. Type /bye to exit.
4. Server mode. Run flm serve qwen3.5:4b. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt qwen3.6-moe:35b-a3b only after the smaller runs succeed, and only with the production service stopped. Watch free -h during the model download and load.

Risks and Open Questions

• Ubuntu 26.04 is unofficial. The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
• The 30 GiB budget is untested at the top end. gpt-oss-20b and 35B-A3B are unknowns. Vendor data covers 32 GB and 96 GB boxes only.
• Concurrent device access is untested. llama-server holds an NPU fd from startup. FLM may or may not open the device alongside it.
• No llama.cpp GGUF path. FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
• The NPU has no production role yet. Nothing here is in service. Treat all NPU content as research until a runbook step lands and is recorded.

Citations

1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM wikipedia_en_all_maxi_2026-02.
9. Live system checks on Dankclaw, 2026-08-16: flm validate output, dpkg -l package list, kernel 7.0.0-29-generic, /proc/3052096/fd device fd inspection.