docs + guard: single 262k slot state, split-limits analysis, cache incident
- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53 (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the KV-per-device rounding rule (round(f x 66)), and why 35 blocks is unreachable (9-FA flip at f~0.538). - PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache entries -> corrupt weights -> degenerate output; fix + signature. - deployment.md + AGENTS.md + index.md: numbers brought current. - scripts/prune-rpc-cache.sh: cron guard capping the worker cache at 40 GiB (oldest first) to prevent the disk-full recurrence.
This commit is contained in:
+16
-7
@@ -143,22 +143,31 @@ curl -sf http://127.0.0.1:7770/health
|
||||
The active service on Dankclaw is the llama.cpp **pipeline-parallel
|
||||
head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
|
||||
alias `frank-default-large` on port 7770. The head splits the tensors
|
||||
`0.40,0.60` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052).
|
||||
`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052):
|
||||
34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split
|
||||
(measured 2026-08-17; see the PP doc for the boundary analysis).
|
||||
Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
|
||||
with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
|
||||
runs **ROCm0** (no `--device` flag; default device selection puts RPC
|
||||
first and picks ROCm for the iGPU). **2 parallel slots** at 200704 context
|
||||
each (pool 401400), KV cache `f16` K / `f16` V, full layer offload
|
||||
(`--gpu-layers all`, `--fit off`), batch 8192 / ubatch 256, `--direct-io`.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool
|
||||
262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per
|
||||
device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192
|
||||
/ ubatch 256, `--direct-io`. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-pp-head.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
|
||||
> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
|
||||
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
|
||||
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
|
||||
> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
|
||||
> output; fixed by clearing the cache + freeing the disk. Guard:
|
||||
> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).
|
||||
|
||||
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
|
||||
> production as `frank-default-large` (renamed from `frank-pp-head`);
|
||||
> the standalone single-node service retired. MTP was re-enabled at
|
||||
> `n_max 4` over the ngram-mod fallback. The pool shrank from 524288 to
|
||||
> 401400 (2 slots x 200700) to fit the split KV budget. See
|
||||
> `n_max 4` over the ngram-mod fallback. See
|
||||
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
|
||||
|
||||
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
|
||||
|
||||
Reference in New Issue
Block a user