diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index c53cade..dd1155c 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -177,6 +177,22 @@ Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_con Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested. +## Parallel Streams (measured 2026-08-16) + +The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. + +Concurrent-request test, gpt-oss-20b, 128-token cap per stream: + +| Streams | Wall s | Aggregate t/s | Per-stream wall s | +|--------:|-------:|--------------:|-------------------| +| 1 | 10.6 | 12.1 | 10.6 | +| 2 | 21.1 | 12.0 | 10.8 / 21.1 | +| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 | + +The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency. + +Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots. + ## Test Runbook Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.