diff --git a/AGENTS.md b/AGENTS.md index 72f97dc..9736bc0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,16 +7,19 @@ local machines. Topology (2026-08-18): | Target | Host | Runtime | Service | Model | Port | |--------|------|---------|---------|-------|------| -| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-small | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView | 52625 | +| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 256K context / 2 slots | 52625 | The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM -NPU endpoint. The RTX 3090 runs the whole 27B model standalone -(no RPC worker). The 2026-08-18 swap retired the vLLM service on Dankclaw; -frank-default-small now runs the CIRU ROCmFPX llama.cpp runtime on the -Strix Halo iGPU (gfx1151). See docs/ciru-activefpx-promptforge.md. -The old scripts live in `old/scripts/` for reference; they describe the -retired PP topology. +NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the +CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151). +A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B +A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small` +(Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is +now `frank-default-large` (CIRU Qwen3.8-27B, 256K context / 2 slots). The +retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See +docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/` +for reference; they describe the retired PP topology. **Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its @@ -33,8 +36,8 @@ when the bench finishes. Rules: 1. Stop the service before any benchmark: - `sudo systemctl stop frank-default-large` (Dankserver) or - `sudo systemctl stop frank-default-small` (Dankclaw). Use the password + `sudo systemctl stop frank-default-small` (Dankserver) or + `sudo systemctl stop frank-default-large` (Dankclaw). Use the password `Assface.3241` with `sudo -S` when no tty is available. 2. Run the bench on a scratch port (never the service ports 7770/52625). 3. Restart the service when the bench completes: @@ -83,9 +86,11 @@ graph LR ## Runtime & Troubleshooting -### FastMTP patched llama.cpp (Dankserver) +### FastMTP patched llama.cpp (retired, 2026-08-18) -The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit +Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 +(files in `.AGENTTRASH/`). The build notes below are kept for reference. +The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit `4df29be4f4c3673f428170fda944a5b19f743bb8` plus the `HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at `/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with @@ -163,7 +168,7 @@ graph TD - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '' | sudo -S `. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. -- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw CIRU ~5-10 minutes (34.9 GiB ActiveFPX payload). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`). +- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`). - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md). diff --git a/docs/deployment.md b/docs/deployment.md index 4c20696..c9ce99a 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView) | frank-default-small | 52625 | +| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 256K ctx / 2 slots) | frank-default-large | 52625 | ## Workflow @@ -72,65 +72,60 @@ Health endpoints: ## Manual Update -### Update Dankserver (frank-default-large) +### Update Dankserver (frank-default-small) Run these commands on Dankserver: ``` cd /home/dank/vLLM git pull origin main -sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service +sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service sudo systemctl daemon-reload -sudo systemctl restart frank-default-large +sudo systemctl restart frank-default-small curl -sf http://127.0.0.1:7770/health ``` -### Update Dankclaw (frank-default-small) +### Update Dankclaw (frank-default-large) Run these commands on Dankclaw: ``` cd /home/dank/vllmScripts git pull origin main -sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service +sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service sudo systemctl daemon-reload -sudo systemctl restart frank-default-small +sudo systemctl restart frank-default-large curl -sf http://127.0.0.1:52625/health ``` ## Service Units -### Dankserver — frank-default-large +### Dankserver — frank-default-small -The active service on Dankserver is llama.cpp serving -`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias -`frank-default-large` on port 7770. It runs the patched FastMTP build -(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be + -the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type -draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the -ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16, -batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is +The active service on Dankserver is upstream llama.cpp (b10121) serving +`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the +alias `frank-default-small` on port 7770. It runs the stock +`/home/dank/llama.cpp/build/bin/llama-server` binary with the 36 MoE +experts on CPU (`--n-cpu-moe 36`) and 7 CPU threads, ngram-mod +speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both +K and V, context 700000 (4 slots of ~175K each), batch 8192 / ubatch 3072, +`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It +launches through `scripts/serve-frank-small.sh`, which reads tunables from +`scripts/frank-default-small.env`. + +### Dankclaw — frank-default-large + +The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving +`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN +companions) under the alias `frank-default-large` on port 52625. It runs +`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm +7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP +(`--spec-type draft-mtp`), f16 KV for both K and V, context 262144 across +2 parallel slots, batch 2048 / ubatch 2048, 32 context checkpoints, 8 GiB +prompt cache, and reasoning in deepseek format. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-frank-large.sh`, which reads tunables from -`scripts/frank-default-large.env`. - -### Dankclaw — frank-default-small - -The active service on Dankclaw is vLLM (ROCm) serving -`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name -`frank-default-small` on port 52625. Full 262144 context, unquantized KV -cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram -speculative decoding, `--reasoning-parser qwen3`, tool calling -(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom -chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and -prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet -+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the -flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends -the same system prompt + tool schema every turn), cutting follow-up TTFT -roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The -unit file is `scripts/frank-default-small.service`. It launches through -`scripts/serve-frank-small.sh`, which reads tunables from -`scripts/frank-default-small.env`. +`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]]. ## Validation @@ -139,7 +134,7 @@ The deploy script checks the health endpoint after a restart. For deeper checks, ``` curl -s http://127.0.0.1:7770/v1/chat/completions \ -H "Content-Type: application/json" \ - -d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}' + -d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}' ``` ## Troubleshooting diff --git a/docs/index.md b/docs/index.md index c29d8e3..6cc1842 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,8 @@ # vLLM Scripts — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-08-18): llama.cpp FastMTP on Dankserver and CIRU ROCmFPX llama.cpp -on Dankclaw. +(2026-08-18): upstream llama.cpp on Dankserver (frank-default-small) and +CIRU ROCmFPX llama.cpp on Dankclaw (frank-default-large). ## Deployment Topology @@ -10,19 +10,19 @@ on Dankclaw. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] A[RTX 3090 24GB + 62GB RAM] - B[llama.cpp FastMTP -frank-default-large, port 7770] - C[Qwen3.8-27B-Aggressive IQ4_XS -FastMTP 32K draft + ngram-mod, 128K ctx] + B[upstream llama.cpp b10121 +frank-default-small, port 7770] + C[Qwen3.6-35B-A3B-Uncensored Q6_K_P +ngram-mod, 36 MoE experts on CPU, 700K ctx] A --> B --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] - D[gfx1151 iGPU unified memory] + D[gfx1151 iGPU 96GB unified memory] E[CIRU ROCmFPX llama.cpp -frank-default-small, port 52625] +frank-default-large, port 52625] F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge -64K ctx, native MTP, PromptForge DualView] +256K ctx / 2 slots, native MTP, PromptForge DualView] D --> E --> F end @@ -42,21 +42,22 @@ frank-default-small, port 52625] - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw - (frank-default-small) -- [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large) -- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-small) + (frank-default-large) +- [[Dankserver]] — the upstream llama.cpp box (frank-default-small) +- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-large) ## CIRU Reference (Dankclaw) - [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build, - and the frank-default-small service + and the frank-default-large service ## vLLM Reference (historical) - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware - notes and sizing (the model family formerly served by frank-default-small) + notes and sizing (the model family now served by frank-default-small as + Q6_K_P) ## Memory Management