frank-primary: four full windows + Restart=always (runtime survivability)
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but left 0 bytes free at runtime: a real client request (995-token prompt, max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1 with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'. Boot is not proof of runtime survivability. Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per rank for activations/graph capture. Unit: Restart=always instead of on-failure — a runtime worker fault makes vLLM's API server exit clean, so ExecStart returned 0 and on-failure never fired, leaving the endpoint down until a human noticed. Also StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
1 parent
bd311bf591
commit
7da504a651
7 files changed
+82
-41
No files matched your search
@@ -722,8 +722,22 @@
|
||||
<= 262,144) and
|
||||
`preserve_thinking` moved from a bare top-level body field to
|
||||
`chat_template_kwargs` (the form vLLM actually reads).
|
||||
- Production bring-up: the winner (241,664) boots and serves on :7770 —
|
||||
engine log verified (KV 732,359 tokens, 3.03x concurrency, MNBT 1024,
|
||||
gather 32x64). Rollback target for the old fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001`; the 237,568 fallback is documented in
|
||||
the runbook.
|
||||
- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
|
||||
passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
|
||||
at runtime; a real client request (995-token prompt, `max_tokens=261149`,
|
||||
sampling params straight from models.yml) killed Worker_TP1 with
|
||||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
|
||||
the engine shut down. Boot is not proof of runtime survivability — after a
|
||||
knife-edge shape change, send a large-`max_tokens` request before calling it
|
||||
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
|
||||
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
|
||||
- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
|
||||
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
|
||||
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
|
||||
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
|
||||
down until a human noticed. An explicit `systemctl stop` still wins, so
|
||||
bench windows and deploys are unaffected.
|
||||
- Production bring-up: the shape boots and serves on :7770 — engine log
|
||||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||||
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
|
||||
Reference in new issue
Block a user