frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
Frank committed 2026-10-01 04:14:25 -07:00
1 parent bd311bf591
commit 7da504a651
7 files changed
+82 -41

No files matched your search

+4 -4
View File
@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
(the model's full native window)**, fp8 KV (6 full-length contexts do not
fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
(the model's full native window)**, fp8 KV (5 boot but leave no runtime
headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
@@ -32,7 +32,7 @@ gfx1201, 32 GiB each]
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
podman run, frank-primary, port 7770]
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
TP=2, expert offload automatic, 5 x 262144 ctx,
TP=2, expert offload automatic, 4 x 262144 ctx,
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
end