From d48227c7f715163b5b7032e19cd9b7eb6b58ca0f Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 16 Aug 2026 12:57:44 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20gpt-oss-20b=20landed=20on=20the=20Dankc?= =?UTF-8?q?law=20NPU=20=E2=80=94=2019.3=20t/s=20decode=20measured,=20memlo?= =?UTF-8?q?ck=20fix,=20serve=20state=20(2026-08-16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/FastFlowLM NPU Inference.md | 35 ++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index 23f3f74..c53cade 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -21,7 +21,8 @@ FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 ch - Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small. - Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw. - Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge). -- One setup blocker remains: the memlock limit. Fix it before any test. +- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results. +- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`. ## What FastFlowLM Is @@ -148,18 +149,33 @@ The vendor also claims "over 10x more power-efficient" versus GPU. That is a mar | Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. | | Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. | | Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. | -| gpt-oss-20b (MoE) | At the edge. 15.1 GB against a ~15 GiB cap. Vendor notes "32 GB machines may work inconsistently". | +| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. | Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either. ## Install State on Dankclaw (verified 2026-08-16) -- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. +- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. A newer release (v1.0.1) exists. Not yet upgraded. - `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed. -- `/dev/accel/accel0` present. `flm validate` passes except memlock. -- No models downloaded yet (`~/.config/flm/` empty). -- No FLM process running. -- The `llama-server` (frank-pp-head) holds fd 4 open on `/dev/accel/accel0` since its own start at 10:43. This is device enumeration at startup. It does not compute on the NPU. Verify FLM can open the device while llama-server runs. +- `/dev/accel/accel0` present. `flm validate` passes fully. +- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed. +- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`. +- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot. +- The `llama-server` (frank-pp-head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. + +## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU) + +Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB. + +| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % | +|-----|-----------:|----------:|-------:|------------:|-----------:|---------:| +| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 | +| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 | +| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a | + +Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client. + +Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested. ## Test Runbook @@ -196,8 +212,9 @@ graph TD ## Risks and Open Questions - **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path. -- **The 30 GiB budget is untested at the top end.** gpt-oss-20b and 35B-A3B are unknowns. Vendor data covers 32 GB and 96 GB boxes only. -- **Concurrent device access is untested.** llama-server holds an NPU fd from startup. FLM may or may not open the device alongside it. +- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only. +- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step. +- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this. - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. - **The NPU has no production role yet.** Nothing here is in service. Treat all NPU content as research until a runbook step lands and is recorded.