dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
+39 -8
View File
@@ -15,10 +15,21 @@ timestamp: 2026-09-03
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
>
> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
> (`ss -ltn`, read 2026-10-18). The runtime is still installed
> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot-llama-prism-bonsai2-pq20]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.
## Verdict (2026-08-16)
@@ -46,7 +57,7 @@ Key properties, per the project README and docs:
## NPU Hardware on Dankclaw
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool.
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", [amd.com product page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents **121 GiB** today (`free -g` total 121, read 2026-10-18).
Live verification on Dankclaw, 2026-08-16:
@@ -60,9 +71,9 @@ $ flm validate
[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity.
```
- Kernel 7.0.0-29-generic ships the `amdxdna` driver in-tree. The driver is loaded.
- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
@@ -78,6 +89,11 @@ Budget math for this fleet:
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
2026-10-18 re-read: Dankclaw's OS now reports **121 GiB** (`free -g`) and
Dankserver **246 GiB** (`free -g`), so the half-of-RAM rule would price the NPU
budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows
are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed.
Weight sizes (approximate, Q4-class):
| Model | Quant | Weights | Fits ~15 GiB budget? |
@@ -162,7 +178,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
- `/dev/accel/accel0` present. `flm validate` passes fully.
- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
@@ -241,7 +257,14 @@ Consequence: the NPU endpoint is a single lane. It suits low-frequency backgroun
## Test Runbook
Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived. Do not run large model loads while the production service holds
most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading,
**121 GiB** today (`free -g`, read 2026-10-18). Check free RAM first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
the box.
```mermaid
graph TD
@@ -387,10 +410,18 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
of Dankclaw killed it. The fleet law today is the opposite: every deployed
container/service ships a boot-persistent, self-healing unit, and the retired
NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
cannot come back at boot.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
`amd_iommu=off` decision that day and the unit is masked today. The remaining
NPU content (benchmarks, runbook steps) is research on a device that is again
usable since 2026-10-18 but serves nothing.
## Citations