dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: four full windows + Restart=always (runtime survivability)

Browse Source 
Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.
This commit is contained in:
Frank
2026-10-01 04:14:25 -07:00
parent bd311bf591
commit 7da504a651
7 changed files with 82 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_primary_unit.bats
+17 -12
View File
@@ -42,7 +42,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# dank session's own hard limit is 8 MiB and cannot be raised without
# restarting user@1000.service).
grep -q '^WantedBy=multi-user.target$' "$UNIT"
grep -q '^Restart=on-failure$' "$UNIT"
# Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
# server exit CLEAN, so ExecStart returns 0 and on-failure never fires
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
grep -q '^Restart=always$' "$UNIT"
! grep -q '^Restart=on-failure$' "$UNIT"
grep -q '^RestartSec=' "$UNIT"
grep -q '^User=dank$' "$UNIT"
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
@@ -75,19 +79,20 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "the qualified engine shape is in the env file" {
# 2026-10-01 night ladder (Frank's method). Two contracts were searched:
# SEQS=3 bf16 topped out at 241664 (262144/245760 OOMed at graph capture;
# KV 732,359 tokens / 3.03x). SEQS=6 fp8 did NOT fit — 6 x 241664 tried
# 10.70 GiB against 9.18 GiB free, so six full-length contexts are
# impossible here. SEQS=5 fp8 at the model's FULL native 262144 window
# PASSED on the first rung and is DEPLOYED: KV 1,350,182 tokens,
# Maximum concurrency 5.15x, 39,462 tokens of margin over the
# 1,310,720-token contract. UTIL stays 0.98 (lowering it shrinks the
# engine's own budget instead of buying slack).
# 2026-10-01 night ladder (Frank's method), then a runtime-survivability
# correction. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
# SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
# free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
# bytes free at runtime and a real client request (995-token prompt with
# max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
# ... 0 bytes free`. FOUR slots at the full native 262144 window is the
# deployed shape: ~1.93 GiB more headroom per rank for
# activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
# own budget instead of buying slack).
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=5$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=4$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=10388766720$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=8311013376$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"