docs: NPU endpoint now serves qwythos:9b (2026-08-17 swap, FLM 1.0.1)
- AGENTS.md + deployment.md target maps: qwen3.6-moe:35b-a3b -> qwythos:9b
- FLM doc: FLM 1.0.1 upgrade record, flm-add install state, model swap
paragraph (commit c85ff6c), FLM_CONFIG_PATH/FLM_XCLBIN_PATH requirement
- pi client entry note: id qwythos:9b, text-only input, developer-role compat kept
This commit is contained in:
Frank
@@ -10,8 +10,9 @@ UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
|
||||
3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
|
||||
7770.
|
||||
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
|
||||
serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx
|
||||
(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is
|
||||
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
|
||||
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
|
||||
`qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is
|
||||
stopped). See [docs/deployment.md](docs/deployment.md) for the deployment
|
||||
process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the
|
||||
spec history and current production state, and
|
||||
@@ -81,7 +82,7 @@ Target map:
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwen3.6-moe:35b-a3b, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 |
|
||||
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 |
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
@@ -155,11 +155,12 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
|
||||
|
||||
## Install State on Dankclaw (verified 2026-08-16)
|
||||
|
||||
- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. A newer release (v1.0.1) exists. Not yet upgraded.
|
||||
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
|
||||
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
|
||||
- `/dev/accel/accel0` present. `flm validate` passes fully.
|
||||
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
|
||||
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
|
||||
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
|
||||
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
|
||||
- The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
|
||||
@@ -205,6 +206,8 @@ The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16,
|
||||
|
||||
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
|
||||
|
||||
Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.
|
||||
|
||||
## Parallel Streams (measured 2026-08-16)
|
||||
|
||||
The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwen3.6-moe:35b-a3b) | frank-default-small | 52625 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
|
||||
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
|
||||
|
||||
The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`.
|
||||
|
||||
Reference in New Issue
Block a user