frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
Frank committed 2026-10-01 04:14:25 -07:00
1 parent bd311bf591
commit 7da504a651
7 files changed
+82 -41

No files matched your search

+12 -11
View File
@@ -60,18 +60,19 @@ FRANK_PRIMARY_PUBLISH=7770:7770
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
# SEQS 5 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape (Frank's call: five
# contexts of the model's FULL native window beats six shortened ones).
# Measured: "GPU KV cache size: 1,350,182 tokens, Maximum concurrency for
# 262,144 tokens per request: 5.15x" — 39,462 tokens of margin over the
# 1,310,720-token contract.
# The night's ladder, both contracts: at SEQS=3 bf16, 262144/245760 OOMed at
# graph capture and 241664 was max possible (KV 732,359 / 3.03x). At SEQS=6
# fp8, 6 x 241664 OOMed (tried 10.70 GiB with 9.18 GiB free) — six
# full-length contexts do NOT fit on this box; five do.
# SEQS 4 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape: four contexts of
# the model's FULL native window, with VRAM headroom the five-slot shape did
# not have. Measured 2026-10-01: 5 x 262144 booted (KV 1,350,182 tokens /
# 5.15x) but left ~0 bytes free at runtime, and a real client request
# (995-token prompt, max_tokens=261149) killed the worker with
# `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free`.
# Dropping to four slots returns ~1.93 GiB per rank for activations/graphs.
# At SEQS=3 bf16, 262144/245760 also OOMed at boot and 241664 was max
# possible; at SEQS=6 fp8, six full-length contexts do NOT fit at all
# (6 x 241664 asked 10.70 GiB against 9.18 GiB free).
FRANK_PRIMARY_MAXLEN=262144
FRANK_PRIMARY_SEQS=5
FRANK_PRIMARY_KV_BYTES=10388766720
FRANK_PRIMARY_SEQS=4
FRANK_PRIMARY_KV_BYTES=8311013376
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=1024
FRANK_PRIMARY_GATHER_CHUNKS=32