From 0799fc5eb4f9eeff3ba8c45b1c65945fabeb5bb3 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 16 Aug 2026 12:29:11 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20FastFlowLM=20NPU=20research=20=E2=80=94?= =?UTF-8?q?=20AMD=20NPU=20runtime,=2050=20TOPS=20Strix=20Halo=20fit=20anal?= =?UTF-8?q?ysis,=20verified=20install=20state,=20test=20runbook=20(2026-08?= =?UTF-8?q?-16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/Dankclaw System Specs.md | 15 +++ docs/FastFlowLM NPU Inference.md | 214 +++++++++++++++++++++++++++++++ docs/index.md | 1 + 3 files changed, 230 insertions(+) create mode 100644 docs/FastFlowLM NPU Inference.md diff --git a/docs/Dankclaw System Specs.md b/docs/Dankclaw System Specs.md index bafd889..a913cea 100644 --- a/docs/Dankclaw System Specs.md +++ b/docs/Dankclaw System Specs.md @@ -26,6 +26,20 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit | GPU-Visible Memory | 96 GiB (driver VRAM ceiling, overcommitted) | | Physical RAM | 30 GiB (shared CPU + GPU) | +### NPU + +| Property | Value | +|----------|-------| +| Model | AMD XDNA2 (Ryzen AI) | +| Rating | 50 TOPS | +| Device | /dev/accel/accel0 | +| Driver | amdxdna 0.7 (in-tree, kernel 7.0.0-29-generic) | +| Firmware | 1.1.2.65 | +| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16) | +| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] | + +The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM for NPU use (~15 GiB here). It is idle in production. + ### ROCm | Property | Value | @@ -81,3 +95,4 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit - [[Shared Memory Architecture Dankclaw]] - [[RAM CPU Offloading in VLLM]] - [[Running Qwen3.6 27B FP8 on Dankclaw]] +- [[FastFlowLM NPU Inference]] diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md new file mode 100644 index 0000000..23f3f74 --- /dev/null +++ b/docs/FastFlowLM NPU Inference.md @@ -0,0 +1,214 @@ +--- +type: research +title: FastFlowLM NPU Inference +description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook. +tags: + - fastflowlm + - npu + - xdna2 + - dankclaw + - research +timestamp: 2026-08-16 +--- + +# FastFlowLM NPU Inference + +FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw and the NPU validates, but nothing serves from it yet. + +## Verdict (2026-08-16) + +- The NPU is a complement, not a replacement. It fits dense models up to about 9B. +- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small. +- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw. +- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge). +- One setup blocker remains: the memlock limit. Fix it before any test. + +## What FastFlowLM Is + +FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0. + +Key properties, per the project README and docs: + +- Runs fully on the NPU. No GPU or CPU load during inference. +- CLI mode (`flm run `) and server mode (`flm serve `). +- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625. +- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face. +- Context lengths up to 256k tokens on listed models. +- Vision, audio, embedding, and MoE model support. +- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use. +- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`. + +## NPU Hardware on Dankclaw + +Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool. + +Live verification on Dankclaw, 2026-08-16: + +``` +$ flm validate +[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json +[Linux] Kernel: 7.0.0-29-generic +[Linux] NPU: /dev/accel/accel0 with 8 columns +[Linux] NPU FW Version: 1.1.2.65 +[Linux] amdxdna version: 0.7 +[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity. +``` + +- Kernel 7.0.0-29-generic ships the `amdxdna` driver in-tree. The driver is loaded. +- Firmware 1.1.2.65 meets the 1.1.0.0 minimum. +- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed. +- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`. +- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working". + +## Memory Model + +The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half. + +Budget math for this fleet: + +| Machine | RAM | NPU budget (approx. half) | +|---------|-----|---------------------------| +| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB | +| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU | +| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB | + +Weight sizes (approximate, Q4-class): + +| Model | Quant | Weights | Fits ~15 GiB budget? | +|-------|-------|---------|----------------------| +| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes | +| Qwen3-8B | Q4_1 | ~5 GB | yes | +| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge | +| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB | +| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists | + +Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run. + +## Supported Models (Qwen family, plus gpt-oss) + +All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima. + +| Tag | Base model | Context max | Notes | +|-----|-----------|-------------|-------| +| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle | +| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle | +| `qwen3:4b` | Qwen3-4B | 32k | tool calling | +| `qwen3:8b` | Qwen3-8B | 32k | tool calling | +| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking | +| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling | +| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision | +| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | | +| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision | +| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle | +| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling | +| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling | +| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling | +| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling | +| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM | + +Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA. + +## Published Benchmarks (FLM, vendor numbers) + +All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed. + +Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length: + +| Model | 1k | 2k | 4k | 8k | 16k | 32k | +|-------|-----:|-----:|-----:|-----:|-----:|-----:| +| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 | +| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 | +| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 | +| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 | + +Prefill t/s for the same run: + +| Model | 1k | 2k | 4k | 8k | 16k | 32k | +|-------|-----:|-----:|-----:|-----:|-----:|-----:| +| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 | +| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 | +| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 | +| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 | + +Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**: + +| Metric | 1k | 2k | 4k | 8k | 16k | 32k | +|--------|-----:|-----:|-----:|-----:|-----:|-----:| +| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 | +| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 | + +Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology. + +The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it. + +## Fit to the Frank Program + +| Our model | NPU verdict | +|-----------|-------------| +| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. | +| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. | +| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. | +| gpt-oss-20b (MoE) | At the edge. 15.1 GB against a ~15 GiB cap. Vendor notes "32 GB machines may work inconsistently". | + +Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either. + +## Install State on Dankclaw (verified 2026-08-16) + +- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. +- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed. +- `/dev/accel/accel0` present. `flm validate` passes except memlock. +- No models downloaded yet (`~/.config/flm/` empty). +- No FLM process running. +- The `llama-server` (frank-pp-head) holds fd 4 open on `/dev/accel/accel0` since its own start at 10:43. This is device enumeration at startup. It does not compute on the NPU. Verify FLM can open the device while llama-server runs. + +## Test Runbook + +Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box. + +```mermaid +graph TD + A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine] + B --> C{NPU visible to XRT?} + C -- no --> D[Fix driver or firmware path] + C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose] + E --> F{Generates tokens?} + F -- no --> D + F -- yes --> G[4. flm serve on a scratch port, never 7770] + G --> H[5. Bench decode + prefill t/s] + H --> I{Speed acceptable?} + I -- no --> J[Record results, stop] + I -- yes --> K[6. Attempt gpt-oss:20b] + K --> L{Loads in ~15 GiB?} + L -- no --> J + L -- yes --> M[Bench, record] + M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped] + N --> O[Record, decision] +``` + +1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead. +2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device. +3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit. +4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770. +5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page. +6. Step up in size: 9B, then gpt-oss-20b. +7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load. + +## Risks and Open Questions + +- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path. +- **The 30 GiB budget is untested at the top end.** gpt-oss-20b and 35B-A3B are unknowns. Vendor data covers 32 GB and 96 GB boxes only. +- **Concurrent device access is untested.** llama-server holds an NPU fd from startup. FLM may or may not open the device alongside it. +- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. +- **The NPU has no production role yet.** Nothing here is in service. Treat all NPU content as research until a runbook step lands and is recorded. + +## Citations + +1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16. +2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16. +3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16. +4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16. +5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16. +6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16. +7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html +8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`. +9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection. diff --git a/docs/index.md b/docs/index.md index 158d2da..f731be5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,6 +57,7 @@ Vulkan0 backend, port 7770, PARALLEL=2] - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings +- [[FastFlowLM NPU Inference]] — FastFlowLM (FLM), the AMD NPU-first runtime. The 50 TOPS Strix Halo NPU on Dankclaw: model fit, vendor benchmarks, verified install state, test runbook (2026-08-16) ## Speculative Decoding & Quantization