diff --git a/AGENTS.md b/AGENTS.md index 09c24b9..6095f47 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,25 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankclaw's NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is stopped). See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS). +This project manages deployment scripts for the LLM inference fleet. +Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth +UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the +`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context +(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090 +`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770. +The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) +serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx +(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is +stopped). See [docs/deployment.md](docs/deployment.md) for the deployment +process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the +spec history and current production state, and +[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md) +for the PP runbook. + +**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the +Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its +OpenWebUI consumes the `:7770` endpoint; its Gitea instance hosts this +repository. Cross-links live in both knowledge-base indexes. ## Benchmarking Policy (MANDATORY) diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index 69b4c70..cab1f32 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -5,6 +5,33 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX morning-of runbook. Research background: [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). +## Production state (promoted 2026-08-16) + +This split is the production `frank-default-large` service (renamed from +`frank-pp-head`, commit `48142f5`). Current state, verified against the +live journal (2026-08-16, 15:03 load): + +- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the + 3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU. +- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`. +- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod + fallback (48/64/24). Journal acceptance: MTP mean len 3.00 + (per-position 0.730/0.541/0.411/0.323). +- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before + Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0. +- Unit `frank-default-large.service`, launched through + `serve-pp-head.sh` + `frank-default-large.env`. + +Open questions from the test day, resolved: + +1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC + boundary — see the acceptance figures above. +2. Two parallel slots hold under production load: 150k-token prompts + prefill and generate, slots turn over every few minutes, no RPC + faults observed. +3. Long prompts (~160k tokens) prefill on the ROCm0 side with no + RPC-boundary anomaly in production traffic. + ## Topology ``` diff --git a/docs/deployment.md b/docs/deployment.md index ac34eff..e72bd0b 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -140,23 +140,36 @@ curl -sf http://127.0.0.1:7770/health ### Dankclaw — frank-default-large -The active service on Dankclaw runs llama.cpp serving the unsloth -Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port -7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft -model) for speculative decoding on the **Vulkan0** backend, with **2 -parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on -GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / -ubatch 256. The unit file is `scripts/frank-default-large.service`. It -launches through `scripts/serve-llama-frank-large.sh`, which reads tunables -from `scripts/frank-default-large.env`. +The active service on Dankclaw is the llama.cpp **pipeline-parallel +head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the +alias `frank-default-large` on port 7770. The head splits the tensors +`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052). +Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`) +with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side +runs **ROCm0** (no `--device` flag; default device selection puts RPC +first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context +each (pool 401400), KV cache `f16` K / `f16` V, full layer offload +(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`. +The unit file is `scripts/frank-default-large.service`. It launches through +`scripts/serve-pp-head.sh`, which reads tunables from +`scripts/frank-default-large.env`. + +> Note: 2026-08-16 — the pipeline-parallel split was promoted to +> production as `frank-default-large` (renamed from `frank-pp-head`); +> the standalone single-node service retired. MTP was re-enabled at +> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to +> 401400 (2 slots x 200700) to fit the split KV budget. See +> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook. > Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k > prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) → > `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went > ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster -> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and -> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU -> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw. +> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone +> configuration; the PP head runs ROCm0 by default (HIP registers before +> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench +> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash +> runtime, and the llama-serve/vLLM units on Dankclaw. ### Dankserver — frank-default-small (STOPPED 2026-08-16) @@ -216,3 +229,6 @@ This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ct - [AGENTS.md](../AGENTS.md) — the git workflow rules - [README.md](../README.md) — repository overview - [index.md](index.md) — knowledge base index +- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) — + the sibling Podman app stack on the same hosts; its OpenWebUI consumes + the `:7770` endpoint, and its Gitea instance hosts this repository diff --git a/docs/index.md b/docs/index.md index 95511a4..32b84ef 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,8 @@ # vLLM Scripts — Knowledge Base -Documentation for running vLLM inference on two local machines. +Documentation for the LLM inference fleet on two local machines: +llama.cpp production (PP head + RPC worker), the FastFlowLM NPU endpoint, +plus a vLLM research archive. ## Deployment Topology @@ -22,8 +24,8 @@ split 0.40/0.60 to worker, MTP+ngram-mod] D --> E --> F G[FastFlowLM NPU XDNA2 50 TOPS frank-default-small, port 52625] - H[gemma4-it:e4b -12.2 t/s, 32k ctx, vision] + H[qwen3.6-moe:35b-a3b +12.5-13.1 t/s, 262k ctx, vision] D --> G --> H end @@ -56,7 +58,7 @@ frank-default-small, port 52625] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) -- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15) +- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16) - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small @@ -77,4 +79,10 @@ frank-default-small, port 52625] - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split - - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/ + - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/ + +## Sibling projects + +- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app + stack on the same two hosts. Its OpenWebUI consumes the `:7770` + endpoint; its Gitea instance hosts this repository. diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index c2dfee4..3e5aff6 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -1,10 +1,41 @@ # Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix This page records the speculative-decoding benchmark and the production -tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw -(`frank-default-large`). +tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service +(`frank-default-large`), now a pipeline-parallel head with a Dankserver +worker. -## Result (applied, 2026-08-15: --spec-default) +## Production state (2026-08-16: PP split + MTP) + +Current production, verified against the live process and journal +(2026-08-16, service start 15:03). Topology detail: +[[Pipeline Parallelism/Qwen3.8 PP Deployment]]. + +- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head + passes no `--device` flag. Default selection puts RPC first and picks + ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays + an option via `HEAD_BIN=.../build-vulkan-rpc`. +- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090 + (its maximum under the KV budget), 60% on the iGPU. +- Context: pool 401400, 2 slots. The journal reports + `n_ctx_slot = 200704`. The pi `models.json` declares 200000 + (conservative). +- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4` + (bundled head) with the `--spec-default` ngram-mod fallback (48/64/24). + Journal acceptance over one production window: MTP mean len 3.00 + (per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the + rare fallback (423 drafts in the same window). +- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0 + at load). +- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, + `--gpu-layers all`. + +## Result (superseded 2026-08-16: standalone --spec-default) + +Superseded for topology on 2026-08-16, when the PP split became +production. The ngram-mod preset survives as the fallback layer under +MTP. The Vulkan0 verdict in this section applies to the standalone +configuration only. - Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). - Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24, @@ -153,22 +184,30 @@ that is the long-context attention penalty, not a spec artifact. ## Production Configuration -`scripts/frank-default-large.env`: +`scripts/frank-default-large.env` (live 2026-08-16): ```ini -SPEC_DEFAULT=1 # llama.cpp preset = ngram-mod (48/64/24) -SPEC_TYPE= -DRAFT_FILE= +RPC_SERVERS=192.168.0.235:50052 # Dankserver worker +TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU] +CTX_SIZE=401400 # pool = 2 slots x 200700 +PARALLEL=2 +SPEC_NONE=0 +SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24) +MTP=1 # draft-mtp, bundled head +SPEC_DRAFT_N_MAX=4 +SPEC_NGRAM_MOD_N_MIN=48 +SPEC_NGRAM_MOD_N_MAX=64 +SPEC_NGRAM_MOD_N_MATCH=24 +CACHE_TYPE_K=f16 +CACHE_TYPE_V=f16 BATCH_SIZE=8192 UBATCH_SIZE=256 -CTX_SIZE=524288 -PARALLEL=2 # passes 2x32k parallel stream test (2026-08-15) -DEVICE=Vulkan0 # 48% faster decode at 262k ctx ``` -Window: computed at 262144 per slot (the pi `models.json` contextWindow for -`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is -a pool ceiling and per-slot stays 262144 regardless of the slot count. +There is no `DEVICE` variable. The head must not pass `--device` (the RPC +device would be silently dropped). Live slot context: `n_ctx_slot = +200704`. The pi `models.json` contextWindow for `frank-default-large` +declares 200000, a safe under-estimate of the live slot. ## See Also