frank-primary: four full windows + Restart=always (runtime survivability)
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but left 0 bytes free at runtime: a real client request (995-token prompt, max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1 with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'. Boot is not proof of runtime survivability. Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per rank for activations/graph capture. Unit: Restart=always instead of on-failure — a runtime worker fault makes vLLM's API server exit clean, so ExecStart returned 0 and on-failure never fired, leaving the endpoint down until a human noticed. Also StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
1 parent
bd311bf591
commit
7da504a651
7 files changed
+82
-41
No files matched your search
@@ -4,9 +4,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **5 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (6 full-length contexts do not
|
||||
fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (5 boot but leave no runtime
|
||||
headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
@@ -32,7 +32,7 @@ gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 5 x 262144 ctx,
|
||||
TP=2, expert offload automatic, 4 x 262144 ctx,
|
||||
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
|
||||
Reference in new issue
Block a user