Merge vllmScripts under llm/ (history-preserving import)
The vllmScripts repository (LLM inference fleet: serve scripts, systemd units, env tunables, deployment runbook, research docs) is merged into this repository. Its full history is grafted under llm/ with a two-parent merge commit so no commit is lost. A follow-up commit reorganizes llm/ into the repo layout: llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
This commit is contained in:
Frank committed
commit
918fe4eaec
82 files changed
+10855
No files matched your search
@@ -0,0 +1,268 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||||
|
||||
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
|
||||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
This split is the production `frank-default-large` service. Current
|
||||
state, verified against the live journal and output smoke tests
|
||||
(2026-08-17, load 02:06):
|
||||
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
|
||||
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
|
||||
zero worker impact — measured, not assumed).
|
||||
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
|
||||
layer weight. Maximum for this config; see "Split limits" below.
|
||||
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
|
||||
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
|
||||
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
|
||||
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
|
||||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||||
- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
|
||||
4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
|
||||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||||
Vulkan).
|
||||
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
|
||||
stable; all allocations are upfront).
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
## Service enablement (2026-08-17)
|
||||
|
||||
The PP pair is the only inference service enabled for boot. Both units
|
||||
(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired
|
||||
units are disabled. Restart order: worker first, then head. The Dankserver
|
||||
helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-default-small` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
[[deployment]] for the full boot-policy table.
|
||||
|
||||
## FA-all-quants rebuild (2026-08-17)
|
||||
|
||||
Flash Attention with mismatched K/V cache types requires the
|
||||
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
|
||||
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
|
||||
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
|
||||
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
|
||||
worker (CUDA sm_86) compile the same sources, so the same CMake flag
|
||||
covers both:
|
||||
|
||||
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
cmake --build build -j --target llama-server # dankclaw
|
||||
cmake --build build -j --target ggml-rpc-server # dankserver
|
||||
|
||||
Both hosts at commit 9b05354ec (b10433). The flag was already cached
|
||||
ON in the worker build (Aug 16 objects carry it); the head required a
|
||||
full ggml-hip rebuild. Keep both binaries on the same commit — version
|
||||
skew hangs the RPC handshake.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
|
||||
|
||||
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
|
||||
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
|
||||
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
|
||||
|
||||
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
|
||||
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
|
||||
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
|
||||
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
|
||||
9408 -> OOM by ~0.85 GiB.
|
||||
|
||||
The KV-per-device count = round(f x 66) layers, independent of the
|
||||
weight boundary (same rule as the Q8_K_XL config). FA blocks at
|
||||
3,7,...,63.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
|
||||
|
||||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||||
exactly 1024 MiB, freeing headroom. Measurements:
|
||||
|
||||
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
|
||||
free. Healthy (2026-08-17 00:29 load).
|
||||
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
|
||||
cudaMalloc OOM by ~740 MiB.
|
||||
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
|
||||
The KV-per-device count rounds the split fraction over 66 layers
|
||||
(round(f x 66)), independent of the weight boundary; the 9-FA flip
|
||||
sits at f ~= 0.538, just below the weight threshold for block 35, so
|
||||
the 35-block config is unreachable.
|
||||
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
|
||||
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
|
||||
|
||||
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
|
||||
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
|
||||
includes the MTP head for weights and token_embd for KV — the two
|
||||
boundaries diverge, which is why 0.54 split them apart.
|
||||
|
||||
## Cache incident (2026-08-17): garbage output from a full worker disk
|
||||
|
||||
Symptom: after a split change, the service loaded "healthy" but every
|
||||
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
|
||||
content, degenerate logits). Root cause chain:
|
||||
|
||||
1. The worker RPC tensor cache keys per session. Each model load writes
|
||||
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
|
||||
4434 files / 96 GB.
|
||||
2. The Dankserver root disk filled to 100%. Cache writes started
|
||||
failing, leaving 0-byte entries. A subsequent reload hit corrupt
|
||||
cache entries and the worker served garbage weights for the
|
||||
newly-needed layers without validation.
|
||||
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
|
||||
worker, free the disk, reload the head (full re-stream ~2.5 min).
|
||||
A clean cache and a healthy disk produced correct output immediately
|
||||
(acceptance 0.89 on the first smoke test).
|
||||
|
||||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||||
Never let the cache directory grow unchecked; a full disk silently
|
||||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||||
token output.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
|
||||
llama-server b10433 ggml-rpc-server b10433
|
||||
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
|
||||
model file (31.5 GB) -d CUDA0 -c (tensor cache)
|
||||
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks)
|
||||
Strix: layers 45-64 + lm_head
|
||||
hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
|
||||
measured wire bytes ~40 KB/position/direction (serialized graph
|
||||
dominates); 20.5 MB per ubatch both directions. Details:
|
||||
[[Ethernet Bandwidth Math]] section 0.
|
||||
```
|
||||
|
||||
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
|
||||
the worker, the rest on the head, in order. One hidden-state handoff per
|
||||
token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
|
||||
of step time). PREFILL and the prompt-cache KV snapshot ARE
|
||||
bandwidth-bound: the link runs at line rate for the wire burst of every
|
||||
ubatch and ~100% during the snapshot copy. Measured numbers:
|
||||
[[Ethernet Bandwidth Math]] section 0.
|
||||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||||
so the production b10121 build stays untouched. A version skew hangs the
|
||||
RPC handshake with no error.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Machine | Purpose |
|
||||
|------|---------|---------|
|
||||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||||
| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||||
| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||||
|
||||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||||
|
||||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||||
`llama_prepare_model_devices` use exactly that list; the RPC device is
|
||||
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
|
||||
puts RPC first.
|
||||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||||
decode (measured on this box).
|
||||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||||
with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
|
||||
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
|
||||
I/O.
|
||||
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
|
||||
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
|
||||
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
|
||||
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
|
||||
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
|
||||
the head before load (warm-cache loads crash fast; cold loads survive).
|
||||
|
||||
## Runbook (morning of 2026-08-16)
|
||||
|
||||
Run from any box with ssh access to both (repo path on claw:
|
||||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||||
|
||||
```
|
||||
# 0. one-time: pull the kit on both boxes
|
||||
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
|
||||
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
|
||||
|
||||
# 1. checks (read-only)
|
||||
./scripts/pp-test.sh preflight
|
||||
|
||||
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
|
||||
|
||||
# or step by step:
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
|
||||
./scripts/pp-test.sh verify # layer split + worker VRAM
|
||||
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
|
||||
./scripts/pp-test.sh bench 16 256 1
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||||
```
|
||||
|
||||
Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
|
||||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||||
|
||||
- No spec baseline: `SPEC_NONE=1`
|
||||
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
|
||||
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
|
||||
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
|
||||
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
|
||||
|
||||
Verification checklist after each `start`:
|
||||
|
||||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||||
1,1 split.
|
||||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||||
(production small service + worker share).
|
||||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||||
|
||||
## Expected numbers
|
||||
|
||||
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
|
||||
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
|
||||
|
||||
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
|
||||
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
|
||||
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
|
||||
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
|
||||
|
||||
## Rollback
|
||||
|
||||
`pp-test.sh restore` starts both production services. If a PP test wedges a
|
||||
box: `systemctl restart <unit>` after killing the stray processes
|
||||
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
|
||||
with a pattern matching your own ssh command line).
|
||||
|
||||
## Open questions for the test day
|
||||
|
||||
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
|
||||
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
|
||||
in the head log (`draft acceptance = X`).
|
||||
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
|
||||
serialized per compute — expect the 2-slot speedup to be weaker than
|
||||
single-node. Measure, do not assume.
|
||||
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
|
||||
the ubatch crossing the RPC boundary is the suspect.
|
||||
Reference in new issue
Block a user