docs: gpt-oss-20b landed on the Dankclaw NPU — 19.3 t/s decode measured, memlock fix, serve state (2026-08-16)

docs/FastFlowLM NPU Inference.md

@@ -21,7 +21,8 @@ FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 ch
 - Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
 - Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
 - Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
-- One setup blocker remains: the memlock limit. Fix it before any test.
+- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
+- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.
 
 ## What FastFlowLM Is
 
@@ -148,18 +149,33 @@ The vendor also claims "over 10x more power-efficient" versus GPU. That is a mar
 | Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
 | Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
 | Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
-| gpt-oss-20b (MoE) | At the edge. 15.1 GB against a ~15 GiB cap. Vendor notes "32 GB machines may work inconsistently". |
+| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |
 
 Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.
 
 ## Install State on Dankclaw (verified 2026-08-16)
 
-- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`.
+- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. A newer release (v1.0.1) exists. Not yet upgraded.
 - `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
-- `/dev/accel/accel0` present. `flm validate` passes except memlock.
-- No models downloaded yet (`~/.config/flm/` empty).
-- No FLM process running.
-- The `llama-server` (frank-pp-head) holds fd 4 open on `/dev/accel/accel0` since its own start at 10:43. This is device enumeration at startup. It does not compute on the NPU. Verify FLM can open the device while llama-server runs.
+- `/dev/accel/accel0` present. `flm validate` passes fully.
+- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
+- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
+- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
+- The `llama-server` (frank-pp-head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
+
+## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
+
+Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.
+
+| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
+|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
+| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
+| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
+| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |
+
+Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.
+
+Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
 
 ## Test Runbook
 
@@ -196,8 +212,9 @@ graph TD
 ## Risks and Open Questions
 
 - **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
-- **The 30 GiB budget is untested at the top end.** gpt-oss-20b and 35B-A3B are unknowns. Vendor data covers 32 GB and 96 GB boxes only.
-- **Concurrent device access is untested.** llama-server holds an NPU fd from startup. FLM may or may not open the device alongside it.
+- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
+- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
+- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
 - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
 - **The NPU has no production role yet.** Nothing here is in service. Treat all NPU content as research until a runbook step lands and is recorded.