diff --git a/AGENTS.md b/AGENTS.md index 6dc994c..dd91d40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,8 +10,9 @@ UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the 3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770. The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) -serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx -(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is +serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag +`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously +`qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is stopped). See [docs/deployment.md](docs/deployment.md) for the deployment process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec history and current production state, and @@ -81,7 +82,7 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwen3.6-moe:35b-a3b, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 | ## Runtime & Troubleshooting diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index ec59d3c..e59b853 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -155,11 +155,12 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization, ## Install State on Dankclaw (verified 2026-08-16) -- `fastflowlm 0.9.45-12-ga6bd056` installed (deb). `flm` at `/usr/bin/flm`. A newer release (v1.0.1) exists. Not yet upgraded. +- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0. - `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed. - `/dev/accel/accel0` present. `flm validate` passes fully. - Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed. - `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`. +- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below). - `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot. - The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. @@ -205,6 +206,8 @@ The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it. +Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept. + ## Parallel Streams (measured 2026-08-16) The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. diff --git a/docs/deployment.md b/docs/deployment.md index fb564cd..0076dfd 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwen3.6-moe:35b-a3b) | frank-default-small | 52625 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 | The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`.