docs: llama.cpp stack at production state (2026-08-16) + sibling-project links

- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
2026-08-16 23:19:21 -07:00
parent f5a78db017
commit 7fe271e65d
5 changed files with 139 additions and 31 deletions
@@ -2,7 +2,25 @@

 ## Overview

-This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankclaw's NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is stopped). See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS).
+This project manages deployment scripts for the LLM inference fleet.
+Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
+UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
+`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context
+(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090
+`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770.
+The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
+serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx
+(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is
+stopped). See [docs/deployment.md](docs/deployment.md) for the deployment
+process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the
+spec history and current production state, and
+[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
+for the PP runbook.
+
+**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
+Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
+OpenWebUI consumes the `:7770` endpoint; its Gitea instance hosts this
+repository. Cross-links live in both knowledge-base indexes.

 ## Benchmarking Policy (MANDATORY)

@@ -5,6 +5,33 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
 morning-of runbook. Research background:
 [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).

+## Production state (promoted 2026-08-16)
+
+This split is the production `frank-default-large` service (renamed from
+`frank-pp-head`, commit `48142f5`). Current state, verified against the
+live journal (2026-08-16, 15:03 load):
+
+- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the
+  3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU.
+- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`.
+- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
+  fallback (48/64/24). Journal acceptance: MTP mean len 3.00
+  (per-position 0.730/0.541/0.411/0.323).
+- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
+  Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0.
+- Unit `frank-default-large.service`, launched through
+  `serve-pp-head.sh` + `frank-default-large.env`.
+
+Open questions from the test day, resolved:
+
+1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC
+   boundary — see the acceptance figures above.
+2. Two parallel slots hold under production load: 150k-token prompts
+   prefill and generate, slots turn over every few minutes, no RPC
+   faults observed.
+3. Long prompts (~160k tokens) prefill on the ROCm0 side with no
+   RPC-boundary anomaly in production traffic.
+
 ## Topology

 ```
@@ -140,23 +140,36 @@ curl -sf http://127.0.0.1:7770/health

 ### Dankclaw — frank-default-large

-The active service on Dankclaw runs llama.cpp serving the unsloth
-Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
-7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
-model) for speculative decoding on the **Vulkan0** backend, with **2
-parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
-GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
-ubatch 256. The unit file is `scripts/frank-default-large.service`. It
-launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
-from `scripts/frank-default-large.env`.
+The active service on Dankclaw is the llama.cpp **pipeline-parallel
+head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
+alias `frank-default-large` on port 7770. The head splits the tensors
+`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
+Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
+with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
+runs **ROCm0** (no `--device` flag; default device selection puts RPC
+first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
+each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
+(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
+The unit file is `scripts/frank-default-large.service`. It launches through
+`scripts/serve-pp-head.sh`, which reads tunables from
+`scripts/frank-default-large.env`.
+
+> Note: 2026-08-16 — the pipeline-parallel split was promoted to
+> production as `frank-default-large` (renamed from `frank-pp-head`);
+> the standalone single-node service retired. MTP was re-enabled at
+> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
+> 401400 (2 slots x 200700) to fit the split KV budget. See
+> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.

 > Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
 > prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
 > `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
 > ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
-> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
-> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
-> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
+> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
+> configuration; the PP head runs ROCm0 by default (HIP registers before
+> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
+> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
+> runtime, and the llama-serve/vLLM units on Dankclaw.

 ### Dankserver — frank-default-small (STOPPED 2026-08-16)

@@ -216,3 +229,6 @@ This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ct
 - [AGENTS.md](../AGENTS.md) — the git workflow rules
 - [README.md](../README.md) — repository overview
 - [index.md](index.md) — knowledge base index
+- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
+  the sibling Podman app stack on the same hosts; its OpenWebUI consumes
+  the `:7770` endpoint, and its Gitea instance hosts this repository
@@ -1,6 +1,8 @@
 # vLLM Scripts — Knowledge Base

-Documentation for running vLLM inference on two local machines.
+Documentation for the LLM inference fleet on two local machines:
+llama.cpp production (PP head + RPC worker), the FastFlowLM NPU endpoint,
+plus a vLLM research archive.

 ## Deployment Topology

@@ -22,8 +24,8 @@ split 0.40/0.60 to worker, MTP+ngram-mod]
        D --> E --> F
        G[FastFlowLM NPU XDNA2 50 TOPS
 frank-default-small, port 52625]
-        H[gemma4-it:e4b
-12.2 t/s, 32k ctx, vision]
+        H[qwen3.6-moe:35b-a3b
+12.5-13.1 t/s, 262k ctx, vision]
        D --> G --> H
    end

@@ -56,7 +58,7 @@ frank-default-small, port 52625]
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
 - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
+- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16)
 - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
 - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
 - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
@@ -77,4 +79,10 @@ frank-default-small, port 52625]
  - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
  - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
  - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
-  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
+  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
+
+## Sibling projects
+
+- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app
+  stack on the same two hosts. Its OpenWebUI consumes the `:7770`
+  endpoint; its Gitea instance hosts this repository.
@@ -1,10 +1,41 @@
 # Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix

 This page records the speculative-decoding benchmark and the production
-tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
-(`frank-default-large`).
+tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
+(`frank-default-large`), now a pipeline-parallel head with a Dankserver
+worker.

-## Result (applied, 2026-08-15: --spec-default)
+## Production state (2026-08-16: PP split + MTP)
+
+Current production, verified against the live process and journal
+(2026-08-16, service start 15:03). Topology detail:
+[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
+
+- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
+  passes no `--device` flag. Default selection puts RPC first and picks
+  ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
+  an option via `HEAD_BIN=.../build-vulkan-rpc`.
+- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
+  (its maximum under the KV budget), 60% on the iGPU.
+- Context: pool 401400, 2 slots. The journal reports
+  `n_ctx_slot = 200704`. The pi `models.json` declares 200000
+  (conservative).
+- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
+  (bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
+  Journal acceptance over one production window: MTP mean len 3.00
+  (per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
+  rare fallback (423 drafts in the same window).
+- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
+  at load).
+- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
+  `--gpu-layers all`.
+
+## Result (superseded 2026-08-16: standalone --spec-default)
+
+Superseded for topology on 2026-08-16, when the PP split became
+production. The ngram-mod preset survives as the fallback layer under
+MTP. The Vulkan0 verdict in this section applies to the standalone
+configuration only.

 - Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
 - Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
@@ -153,22 +184,30 @@ that is the long-context attention penalty, not a spec artifact.

 ## Production Configuration

-`scripts/frank-default-large.env`:
+`scripts/frank-default-large.env` (live 2026-08-16):

 ```ini
-SPEC_DEFAULT=1            # llama.cpp preset = ngram-mod (48/64/24)
-SPEC_TYPE=
-DRAFT_FILE=
+RPC_SERVERS=192.168.0.235:50052   # Dankserver worker
+TENSOR_SPLIT=0.40,0.60            # [RPC0, iGPU]
+CTX_SIZE=401400                   # pool = 2 slots x 200700
+PARALLEL=2
+SPEC_NONE=0
+SPEC_DEFAULT=1                    # ngram-mod fallback (48/64/24)
+MTP=1                             # draft-mtp, bundled head
+SPEC_DRAFT_N_MAX=4
+SPEC_NGRAM_MOD_N_MIN=48
+SPEC_NGRAM_MOD_N_MAX=64
+SPEC_NGRAM_MOD_N_MATCH=24
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
 BATCH_SIZE=8192
 UBATCH_SIZE=256
-CTX_SIZE=524288
-PARALLEL=2                # passes 2x32k parallel stream test (2026-08-15)
-DEVICE=Vulkan0            # 48% faster decode at 262k ctx
 ```

-Window: computed at 262144 per slot (the pi `models.json` contextWindow for
-`frank-default-large`).  The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
-a pool ceiling and per-slot stays 262144 regardless of the slot count.
+There is no `DEVICE` variable. The head must not pass `--device` (the RPC
+device would be silently dropped). Live slot context: `n_ctx_slot =
+200704`. The pi `models.json` contextWindow for `frank-default-large`
+declares 200000, a safe under-estimate of the live slot.

 ## See Also