From aa1941d592a9b97ae367db5074d6a4d17f2ca85b Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 17 Aug 2026 01:24:06 -0700 Subject: [PATCH] docs + guard: single 262k slot state, split-limits analysis, cache incident - spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53 (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the KV-per-device rounding rule (round(f x 66)), and why 35 blocks is unreachable (9-FA flip at f~0.538). - PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache entries -> corrupt weights -> degenerate output; fix + signature. - deployment.md + AGENTS.md + index.md: numbers brought current. - scripts/prune-rpc-cache.sh: cron guard capping the worker cache at 40 GiB (oldest first) to prevent the disk-full recurrence. --- AGENTS.md | 7 +- .../Qwen3.8 PP Deployment.md | 73 ++++++++++++++----- docs/deployment.md | 23 ++++-- docs/index.md | 4 +- docs/qwen3.8-spec-matrix.md | 34 +++++---- scripts/prune-rpc-cache.sh | 36 +++++++++ 6 files changed, 132 insertions(+), 45 deletions(-) create mode 100755 scripts/prune-rpc-cache.sh diff --git a/AGENTS.md b/AGENTS.md index 6095f47..6dc994c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,9 +5,10 @@ This project manages deployment scripts for the LLM inference fleet. Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the -`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context -(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090 -`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770. +`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool +262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX +3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port +7770. The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx (swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index cab1f32..0fe5177 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -5,32 +5,69 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX morning-of runbook. Research background: [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). -## Production state (promoted 2026-08-16) +## Production state (2026-08-17: single 262k slot, split 0.53) -This split is the production `frank-default-large` service (renamed from -`frank-pp-head`, commit `48142f5`). Current state, verified against the -live journal (2026-08-16, 15:03 load): +This split is the production `frank-default-large` service. Current +state, verified against the live journal and output smoke test +(2026-08-17, load 01:21): -- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the - 3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU. -- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`. +- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum + provably-safe value; see "Split limits" below for the boundary + analysis and the failed 0.54/0.55 attempts. +- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`. - MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod - fallback (48/64/24). Journal acceptance: MTP mean len 3.00 - (per-position 0.730/0.541/0.411/0.323). + fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17). - Head GPU side: **ROCm0** (no `--device` flag; HIP registers before - Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0. + Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each). +- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free. - Unit `frank-default-large.service`, launched through `serve-pp-head.sh` + `frank-default-large.env`. -Open questions from the test day, resolved: +## Split limits (measured 2026-08-17) -1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC - boundary — see the acceptance figures above. -2. Two parallel slots hold under production load: 150k-token prompts - prefill and generate, slots turn over every few minutes, no RPC - faults observed. -3. Long prompts (~160k tokens) prefill on the ROCm0 side with no - RPC-boundary anomaly in production traffic. +The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks. +With the pool shrunk to 262144, the KV per full-attention layer drops to +exactly 1024 MiB, freeing headroom. Measurements: + +- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB + free. Healthy (2026-08-17 00:29 load). +- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB -> + cudaMalloc OOM by ~740 MiB. +- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM. + The KV-per-device count rounds the split fraction over 66 layers + (round(f x 66)), independent of the weight boundary; the 9-FA flip + sits at f ~= 0.538, just below the weight threshold for block 35, so + the 35-block config is unreachable. +- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every + rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production. + +GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each), +linear blocks 431.3 MB; total layer weight 27720 MB. The split pool +includes the MTP head for weights and token_embd for KV — the two +boundaries diverge, which is why 0.54 split them apart. + +## Cache incident (2026-08-17): garbage output from a full worker disk + +Symptom: after a split change, the service loaded "healthy" but every +completion degenerated to a repeated junk token (e.g. 48 x "/", empty +content, degenerate logits). Root cause chain: + +1. The worker RPC tensor cache keys per session. Each model load writes + a fresh full copy (~15 GB); over ~5 sessions of two models it grew to + 4434 files / 96 GB. +2. The Dankserver root disk filled to 100%. Cache writes started + failing, leaving 0-byte entries. A subsequent reload hit corrupt + cache entries and the worker served garbage weights for the + newly-needed layers without validation. +3. Fix: move the cache to .AGENTTRASH (then empty it), restart the + worker, free the disk, reload the head (full re-stream ~2.5 min). + A clean cache and a healthy disk produced correct output immediately + (acceptance 0.89 on the first smoke test). + +Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). +Never let the cache directory grow unchecked; a full disk silently +corrupts it. Signature to recognize: healthy load + degenerate repeated- +token output. ## Topology diff --git a/docs/deployment.md b/docs/deployment.md index e72bd0b..fb564cd 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -143,22 +143,31 @@ curl -sf http://127.0.0.1:7770/health The active service on Dankclaw is the llama.cpp **pipeline-parallel head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port 7770. The head splits the tensors -`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052). +`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052): +34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split +(measured 2026-08-17; see the PP doc for the boundary analysis). Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`) with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side runs **ROCm0** (no `--device` flag; default device selection puts RPC -first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context -each (pool 401400), KV cache `f16` K / `f16` V, full layer offload -(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`. -The unit file is `scripts/frank-default-large.service`. It launches through +first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool +262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per +device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192 +/ ubatch 256, `--direct-io`. The unit file is +`scripts/frank-default-large.service`. It launches through `scripts/serve-pp-head.sh`, which reads tunables from `scripts/frank-default-large.env`. +> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144 +> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090 +> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full +> Dankserver disk corrupted the worker RPC tensor cache -> degenerate +> output; fixed by clearing the cache + freeing the disk. Guard: +> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap). + > Note: 2026-08-16 — the pipeline-parallel split was promoted to > production as `frank-default-large` (renamed from `frank-pp-head`); > the standalone single-node service retired. MTP was re-enabled at -> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to -> 401400 (2 slots x 200700) to fit the split KV budget. See +> `n_max 4` over the ngram-mod fallback. See > [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook. > Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k diff --git a/docs/index.md b/docs/index.md index 32b84ef..89cc24c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -58,7 +58,7 @@ frank-default-small, port 52625] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) -- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16) +- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 1 slot @ 262144, split 0.53 (2026-08-17) - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small @@ -79,7 +79,7 @@ frank-default-small, port 52625] - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split - - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/ + - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits + cache incident measured 2026-08-17 ## Sibling projects diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index 3e5aff6..f407fe4 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -5,32 +5,36 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service (`frank-default-large`), now a pipeline-parallel head with a Dankserver worker. -## Production state (2026-08-16: PP split + MTP) +## Production state (2026-08-17: single 262k slot, split 0.53) Current production, verified against the live process and journal -(2026-08-16, service start 15:03). Topology detail: +(2026-08-17, load 01:21, output smoke test). Topology detail: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]. - Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head passes no `--device` flag. Default selection puts RPC first and picks ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays an option via `HEAD_BIN=.../build-vulkan-rpc`. -- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090 - (its maximum under the KV budget), 60% on the iGPU. -- Context: pool 401400, 2 slots. The journal reports - `n_ctx_slot = 200704`. The pi `models.json` declares 200000 - (conservative). -- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4` - (bundled head) with the `--spec-default` ngram-mod fallback (48/64/24). - Journal acceptance over one production window: MTP mean len 3.00 - (per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the - rare fallback (423 drafts in the same window). -- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0 - at load). +- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the + 3090, about 52% of the layer weight. This is the maximum provably-safe + value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention + layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction + over 66 layers, independent of the weight boundary; the 9-FA flip sits + at frac ~= 0.538. Boundary analysis: the PP deployment doc. +- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi + `models.json` declares 262144 (contextWindow = maxTokens, house + convention). +- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled + head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke + acceptance 0.83 (50/60, 2026-08-17); full production-window stats in + the superseded section below. +- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device + (8 full-attention layers each; FA at blocks 3,7,...,63). +- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free. - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, `--gpu-layers all`. -## Result (superseded 2026-08-16: standalone --spec-default) +## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40) Superseded for topology on 2026-08-16, when the PP split became production. The ngram-mod preset survives as the fallback layer under diff --git a/scripts/prune-rpc-cache.sh b/scripts/prune-rpc-cache.sh new file mode 100755 index 0000000..48632a2 --- /dev/null +++ b/scripts/prune-rpc-cache.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Bound the ggml-rpc worker tensor cache. +# +# The worker (frank-pp-worker on Dankserver) keys cache entries per +# session, so each model load writes a fresh full copy (~15 GB). Left +# unbounded, the directory filled the root disk to 100% on 2026-08-17; +# failed cache writes left 0-byte entries, and a later reload served +# corrupt weights -> degenerate model output (see the cache incident in +# docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md). +# +# This script prunes the cache to a byte cap, oldest files first. Cache +# entries are derived data: safe to delete. Run from cron on Dankserver. +set -euo pipefail + +CACHE_DIR=/home/dank/.cache/llama.cpp/rpc +MAX_BYTES=$((40 * 1024 * 1024 * 1024)) # 40 GiB + +if [[ ! -d "$CACHE_DIR" ]]; then + exit 0 +fi + +total=$(du -sb "$CACHE_DIR" | cut -f1) +if (( total <= MAX_BYTES )); then + exit 0 +fi + +while (( total > MAX_BYTES )); do + oldest=$(ls -1t "$CACHE_DIR" | tail -1) + if [[ -z "$oldest" ]]; then + break + fi + rm -f -- "$CACHE_DIR/$oldest" + total=$(du -sb "$CACHE_DIR" | cut -f1) +done + +echo "prune-rpc-cache: pruned to $(du -sh "$CACHE_DIR" | cut -f1)"