frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
Frank committed 2026-10-01 04:14:25 -07:00
1 parent bd311bf591
commit 7da504a651
7 files changed
+82 -41

No files matched your search

@@ -278,8 +278,8 @@ is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-01: 5 x 262144, fp8, no spec)
```
FRANK_PRIMARY_MAXLEN=262144 x SEQS 5 (1,310,720 tokens of contract)
FRANK_PRIMARY_KV_BYTES=10388766720 PER RANK (fp8, pool scales with windows)
FRANK_PRIMARY_MAXLEN=262144 x SEQS 4 (1,048,576 tokens of contract)
FRANK_PRIMARY_KV_BYTES=8311013376 PER RANK (fp8, pool scales with windows)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (K and V together)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
@@ -297,10 +297,14 @@ memory environment). The night searched two concurrency contracts:
| 3 × 237568, bf16 | PASS — KV 718,606 tokens |
| **3 × 241664, bf16** | **PASS — KV 732,359 tokens, 3.03x** (max possible at 3 contexts) |
| 6 × 241664, fp8 | **OOM** — tried 10.70 GiB against 9.18 GiB free: six full-length contexts do NOT fit |
| **5 × 262144, fp8** | **PASS on the first rung — KV 1,350,182 tokens, 5.15x; DEPLOYED** (39,462-token margin over the 1,310,720-token contract) |
| 5 × 262144, fp8 | PASS on the first rung — KV 1,350,182 tokens, 5.15x |
| **4 × 262144, fp8** | **DEPLOYED.** Five slots booted but left 0 bytes free at runtime: a real client request (995-token prompt, `max_tokens=261149`) killed the worker (`Tried to allocate 50.00 MiB ... 0 bytes free`). Dropping one slot returns ~1.93 GiB per rank for activations and graph capture |
The final shape is Frank's call: **five contexts of the model's full native
262,144 window** beat six shortened ones. fp8 KV is what buys the capacity —
Boot is NOT proof of runtime survivability: the five-slot shape passed every
boot gate and still died on the first long-generation request. After any
knife-edge shape change, send a request with a large `max_tokens` before
calling it done. The deployed shape is **four contexts of the model's full
native 262,144 window**. fp8 KV is what buys the capacity —
7,926 B/token against bf16's ~13,900-14,047 — at the cost of bf16's ~4 %
decode edge. Six full-length contexts cannot fit at any KV dtype: the 6-seq
graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144