dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: NPU parallelism measured — single-lane FIFO serialization, aggregate t/s flat at N=1/2/4

Browse Source
This commit is contained in:
Frank
2026-08-16 13:12:46 -07:00
parent d48227c7f7
commit 79c913aaa7
1 changed files with 16 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FastFlowLM NPU Inference.md
+16
View File
@@ -177,6 +177,22 @@ Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_con
Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
## Parallel Streams (measured 2026-08-16)
The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
Concurrent-request test, gpt-oss-20b, 128-token cap per stream:
| Streams | Wall s | Aggregate t/s | Per-stream wall s |
|--------:|-------:|--------------:|-------------------|
| 1 | 10.6 | 12.1 | 10.6 |
| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |
The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.
Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
## Test Runbook
Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.