frank-primary: four full windows + Restart=always (runtime survivability)
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but left 0 bytes free at runtime: a real client request (995-token prompt, max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1 with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'. Boot is not proof of runtime survivability. Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per rank for activations/graph capture. Unit: Restart=always instead of on-failure — a runtime worker fault makes vLLM's API server exit clean, so ExecStart returned 0 and on-failure never fired, leaving the endpoint down until a human noticed. Also StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
1 parent
bd311bf591
commit
7da504a651
7 files changed
+82
-41
No files matched your search
+12
-11
@@ -60,18 +60,19 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
|
||||
# SEQS 5 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape (Frank's call: five
|
||||
# contexts of the model's FULL native window beats six shortened ones).
|
||||
# Measured: "GPU KV cache size: 1,350,182 tokens, Maximum concurrency for
|
||||
# 262,144 tokens per request: 5.15x" — 39,462 tokens of margin over the
|
||||
# 1,310,720-token contract.
|
||||
# The night's ladder, both contracts: at SEQS=3 bf16, 262144/245760 OOMed at
|
||||
# graph capture and 241664 was max possible (KV 732,359 / 3.03x). At SEQS=6
|
||||
# fp8, 6 x 241664 OOMed (tried 10.70 GiB with 9.18 GiB free) — six
|
||||
# full-length contexts do NOT fit on this box; five do.
|
||||
# SEQS 4 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape: four contexts of
|
||||
# the model's FULL native window, with VRAM headroom the five-slot shape did
|
||||
# not have. Measured 2026-10-01: 5 x 262144 booted (KV 1,350,182 tokens /
|
||||
# 5.15x) but left ~0 bytes free at runtime, and a real client request
|
||||
# (995-token prompt, max_tokens=261149) killed the worker with
|
||||
# `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free`.
|
||||
# Dropping to four slots returns ~1.93 GiB per rank for activations/graphs.
|
||||
# At SEQS=3 bf16, 262144/245760 also OOMed at boot and 241664 was max
|
||||
# possible; at SEQS=6 fp8, six full-length contexts do NOT fit at all
|
||||
# (6 x 241664 asked 10.70 GiB against 9.18 GiB free).
|
||||
FRANK_PRIMARY_MAXLEN=262144
|
||||
FRANK_PRIMARY_SEQS=5
|
||||
FRANK_PRIMARY_KV_BYTES=10388766720
|
||||
FRANK_PRIMARY_SEQS=4
|
||||
FRANK_PRIMARY_KV_BYTES=8311013376
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=1024
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
|
||||
Reference in new issue
Block a user