diff --git a/AGENTS.md b/AGENTS.md index b20ebe2..09c24b9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS). +This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankclaw's NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is stopped). See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS). ## Benchmarking Policy (MANDATORY) @@ -62,7 +62,7 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, gemma4-it:e4b; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwen3.6-moe:35b-a3b, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 | ## Runtime & Troubleshooting diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index 3bec0e3..8bc8a7d 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -13,7 +13,7 @@ timestamp: 2026-08-16 # FastFlowLM NPU Inference -FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw and the NPU validates, but nothing serves from it yet. +FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-default-small` (port 52625) serves in production. See Promotion. ## Verdict (2026-08-16) @@ -203,6 +203,8 @@ Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`). +Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it. + ## Parallel Streams (measured 2026-08-16) The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. @@ -251,6 +253,26 @@ graph TD 6. Step up in size: 9B, then gpt-oss-20b. 7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load. +## Incident: qwen3.6-moe unreachable (2026-08-16) + +Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window. + +1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`. +2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-default-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-default-small' is not supported.` Each such request also triggered the unknown-tag fallback below. +3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load. + +Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16. + +### Unknown-tag fallback (FLM behavior) + +When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`: + +- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs. +- FLM then loads the 1B model in place of the requested model. +- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error. + +Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model. + ## Risks and Open Questions - **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path. @@ -258,7 +280,7 @@ graph TD - **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step. - **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this. - **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. -- **The NPU has no production role yet.** Nothing here is in service. Treat all NPU content as research until a runbook step lands and is recorded. +- **NPU production role (2026-08-16).** `frank-default-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded. ## Citations diff --git a/docs/deployment.md b/docs/deployment.md index 20733ee..ac34eff 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -8,8 +8,11 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | -| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwen3.6-moe:35b-a3b) | frank-default-small | 52625 | +| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 | + +The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`. ## Workflow @@ -155,7 +158,9 @@ from `scripts/frank-default-large.env`. > the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU > INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw. -### Dankserver — frank-default-small +### Dankserver — frank-default-small (STOPPED 2026-08-16) + +> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history. The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.