docs + guard: single 262k slot state, split-limits analysis, cache incident
- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53 (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the KV-per-device rounding rule (round(f x 66)), and why 35 blocks is unreachable (9-FA flip at f~0.538). - PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache entries -> corrupt weights -> degenerate output; fix + signature. - deployment.md + AGENTS.md + index.md: numbers brought current. - scripts/prune-rpc-cache.sh: cron guard capping the worker cache at 40 GiB (oldest first) to prevent the disk-full recurrence.
This commit is contained in:
Frank
@@ -5,9 +5,10 @@
|
||||
This project manages deployment scripts for the LLM inference fleet.
|
||||
Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
|
||||
UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
|
||||
`--spec-default` ngram-mod fallback, 2 parallel slots at 200704 context
|
||||
(pool 401400), tensors split 0.40/0.60 to the Dankserver RTX 3090
|
||||
`ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port 7770.
|
||||
`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool
|
||||
262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX
|
||||
3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
|
||||
7770.
|
||||
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
|
||||
serves Qwen3.6-35B-A3B as registry tag `qwen3.6-moe:35b-a3b` at 262144 ctx
|
||||
(swapped in 2026-08-16; the old llama.cpp 35B service on Dankserver is
|
||||
|
||||
@@ -5,32 +5,69 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Production state (promoted 2026-08-16)
|
||||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||||
|
||||
This split is the production `frank-default-large` service (renamed from
|
||||
`frank-pp-head`, commit `48142f5`). Current state, verified against the
|
||||
live journal (2026-08-16, 15:03 load):
|
||||
This split is the production `frank-default-large` service. Current
|
||||
state, verified against the live journal and output smoke test
|
||||
(2026-08-17, load 01:21):
|
||||
|
||||
- Split `0.40,0.60` (not the test-day `1,1`): 40% of the layers on the
|
||||
3090 — its maximum under the 2x200700 f16 KV budget — 60% on the iGPU.
|
||||
- Pool `401400`, `PARALLEL=2`, live `n_ctx_slot = 200704`.
|
||||
- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
|
||||
provably-safe value; see "Split limits" below for the boundary
|
||||
analysis and the failed 0.54/0.55 attempts.
|
||||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||||
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
|
||||
fallback (48/64/24). Journal acceptance: MTP mean len 3.00
|
||||
(per-position 0.730/0.541/0.411/0.323).
|
||||
fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
|
||||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||||
Vulkan). KV at load: 15680 MiB ROCm0 + 9408 MiB RPC0.
|
||||
Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
|
||||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
Open questions from the test day, resolved:
|
||||
## Split limits (measured 2026-08-17)
|
||||
|
||||
1. MTP (in the last layer group, iGPU side) drafts cleanly across the RPC
|
||||
boundary — see the acceptance figures above.
|
||||
2. Two parallel slots hold under production load: 150k-token prompts
|
||||
prefill and generate, slots turn over every few minutes, no RPC
|
||||
faults observed.
|
||||
3. Long prompts (~160k tokens) prefill on the ROCm0 side with no
|
||||
RPC-boundary anomaly in production traffic.
|
||||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||||
exactly 1024 MiB, freeing headroom. Measurements:
|
||||
|
||||
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
|
||||
free. Healthy (2026-08-17 00:29 load).
|
||||
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
|
||||
cudaMalloc OOM by ~740 MiB.
|
||||
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
|
||||
The KV-per-device count rounds the split fraction over 66 layers
|
||||
(round(f x 66)), independent of the weight boundary; the 9-FA flip
|
||||
sits at f ~= 0.538, just below the weight threshold for block 35, so
|
||||
the 35-block config is unreachable.
|
||||
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
|
||||
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
|
||||
|
||||
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
|
||||
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
|
||||
includes the MTP head for weights and token_embd for KV — the two
|
||||
boundaries diverge, which is why 0.54 split them apart.
|
||||
|
||||
## Cache incident (2026-08-17): garbage output from a full worker disk
|
||||
|
||||
Symptom: after a split change, the service loaded "healthy" but every
|
||||
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
|
||||
content, degenerate logits). Root cause chain:
|
||||
|
||||
1. The worker RPC tensor cache keys per session. Each model load writes
|
||||
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
|
||||
4434 files / 96 GB.
|
||||
2. The Dankserver root disk filled to 100%. Cache writes started
|
||||
failing, leaving 0-byte entries. A subsequent reload hit corrupt
|
||||
cache entries and the worker served garbage weights for the
|
||||
newly-needed layers without validation.
|
||||
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
|
||||
worker, free the disk, reload the head (full re-stream ~2.5 min).
|
||||
A clean cache and a healthy disk produced correct output immediately
|
||||
(acceptance 0.89 on the first smoke test).
|
||||
|
||||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||||
Never let the cache directory grow unchecked; a full disk silently
|
||||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||||
token output.
|
||||
|
||||
## Topology
|
||||
|
||||
|
||||
+16
-7
@@ -143,22 +143,31 @@ curl -sf http://127.0.0.1:7770/health
|
||||
The active service on Dankclaw is the llama.cpp **pipeline-parallel
|
||||
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
|
||||
alias `frank-default-large` on port 7770. The head splits the tensors
|
||||
`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
|
||||
`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052):
|
||||
34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split
|
||||
(measured 2026-08-17; see the PP doc for the boundary analysis).
|
||||
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
|
||||
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
|
||||
runs **ROCm0** (no `--device` flag; default device selection puts RPC
|
||||
first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
|
||||
each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
|
||||
(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool
|
||||
262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per
|
||||
device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192
|
||||
/ ubatch 256, `--direct-io`. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-pp-head.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
|
||||
> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
|
||||
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
|
||||
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
|
||||
> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
|
||||
> output; fixed by clearing the cache + freeing the disk. Guard:
|
||||
> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).
|
||||
|
||||
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
|
||||
> production as `frank-default-large` (renamed from `frank-pp-head`);
|
||||
> the standalone single-node service retired. MTP was re-enabled at
|
||||
> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
|
||||
> 401400 (2 slots x 200700) to fit the split KV budget. See
|
||||
> `n_max 4` over the ngram-mod fallback. See
|
||||
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
|
||||
|
||||
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
|
||||
|
||||
+2
-2
@@ -58,7 +58,7 @@ frank-default-small, port 52625]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 2 slots @ 200704 (2026-08-16)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 1 slot @ 262144, split 0.53 (2026-08-17)
|
||||
- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
|
||||
@@ -79,7 +79,7 @@ frank-default-small, port 52625]
|
||||
- [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
|
||||
- [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
|
||||
- [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
|
||||
- [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment (promoted 2026-08-16) + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
|
||||
- [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits + cache incident measured 2026-08-17
|
||||
|
||||
## Sibling projects
|
||||
|
||||
|
||||
+19
-15
@@ -5,32 +5,36 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
||||
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
||||
worker.
|
||||
|
||||
## Production state (2026-08-16: PP split + MTP)
|
||||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-16, service start 15:03). Topology detail:
|
||||
(2026-08-17, load 01:21, output smoke test). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
|
||||
(its maximum under the KV budget), 60% on the iGPU.
|
||||
- Context: pool 401400, 2 slots. The journal reports
|
||||
`n_ctx_slot = 200704`. The pi `models.json` declares 200000
|
||||
(conservative).
|
||||
- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
|
||||
(bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
|
||||
Journal acceptance over one production window: MTP mean len 3.00
|
||||
(per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
|
||||
rare fallback (423 drafts in the same window).
|
||||
- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
|
||||
at load).
|
||||
- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
|
||||
3090, about 52% of the layer weight. This is the maximum provably-safe
|
||||
value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
|
||||
layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
|
||||
over 66 layers, independent of the weight boundary; the 9-FA flip sits
|
||||
at frac ~= 0.538. Boundary analysis: the PP deployment doc.
|
||||
- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
|
||||
`models.json` declares 262144 (contextWindow = maxTokens, house
|
||||
convention).
|
||||
- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
|
||||
head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
|
||||
acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
|
||||
the superseded section below.
|
||||
- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
|
||||
(8 full-attention layers each; FA at blocks 3,7,...,63).
|
||||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## Result (superseded 2026-08-16: standalone --spec-default)
|
||||
## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
|
||||
|
||||
Superseded for topology on 2026-08-16, when the PP split became
|
||||
production. The ngram-mod preset survives as the fallback layer under
|
||||
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bound the ggml-rpc worker tensor cache.
|
||||
#
|
||||
# The worker (frank-pp-worker on Dankserver) keys cache entries per
|
||||
# session, so each model load writes a fresh full copy (~15 GB). Left
|
||||
# unbounded, the directory filled the root disk to 100% on 2026-08-17;
|
||||
# failed cache writes left 0-byte entries, and a later reload served
|
||||
# corrupt weights -> degenerate model output (see the cache incident in
|
||||
# docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md).
|
||||
#
|
||||
# This script prunes the cache to a byte cap, oldest files first. Cache
|
||||
# entries are derived data: safe to delete. Run from cron on Dankserver.
|
||||
set -euo pipefail
|
||||
|
||||
CACHE_DIR=/home/dank/.cache/llama.cpp/rpc
|
||||
MAX_BYTES=$((40 * 1024 * 1024 * 1024)) # 40 GiB
|
||||
|
||||
if [[ ! -d "$CACHE_DIR" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
total=$(du -sb "$CACHE_DIR" | cut -f1)
|
||||
if (( total <= MAX_BYTES )); then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
while (( total > MAX_BYTES )); do
|
||||
oldest=$(ls -1t "$CACHE_DIR" | tail -1)
|
||||
if [[ -z "$oldest" ]]; then
|
||||
break
|
||||
fi
|
||||
rm -f -- "$CACHE_DIR/$oldest"
|
||||
total=$(du -sb "$CACHE_DIR" | cut -f1)
|
||||
done
|
||||
|
||||
echo "prune-rpc-cache: pruned to $(du -sh "$CACHE_DIR" | cut -f1)"
|
||||
Reference in New Issue
Block a user