frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
Frank committed 2026-10-01 04:14:25 -07:00
1 parent bd311bf591
commit 7da504a651
7 files changed
+82 -41

No files matched your search

+19 -5
View File
@@ -722,8 +722,22 @@
<= 262,144) and
`preserve_thinking` moved from a bare top-level body field to
`chat_template_kwargs` (the form vLLM actually reads).
- Production bring-up: the winner (241,664) boots and serves on :7770 —
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
the runbook.
- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
at runtime; a real client request (995-token prompt, `max_tokens=261149`,
sampling params straight from models.yml) killed Worker_TP1 with
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
the engine shut down. Boot is not proof of runtime survivability — after a
knife-edge shape change, send a large-`max_tokens` request before calling it
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
down until a human noticed. An explicit `systemctl stop` still wins, so
bench windows and deploys are unaffected.
- Production bring-up: the shape boots and serves on :7770 — engine log
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
contract probes (reasoning split, tool_calls). Rollback target for the
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.