frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.

scripts/frank-primary.env

@@ -60,18 +60,19 @@ FRANK_PRIMARY_PUBLISH=7770:7770
 FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
 
 # --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
-# SEQS 5 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape (Frank's call: five
-# contexts of the model's FULL native window beats six shortened ones).
-# Measured: "GPU KV cache size: 1,350,182 tokens, Maximum concurrency for
-# 262,144 tokens per request: 5.15x" — 39,462 tokens of margin over the
-# 1,310,720-token contract.
-# The night's ladder, both contracts: at SEQS=3 bf16, 262144/245760 OOMed at
-# graph capture and 241664 was max possible (KV 732,359 / 3.03x). At SEQS=6
-# fp8, 6 x 241664 OOMed (tried 10.70 GiB with 9.18 GiB free) — six
-# full-length contexts do NOT fit on this box; five do.
+# SEQS 4 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape: four contexts of
+# the model's FULL native window, with VRAM headroom the five-slot shape did
+# not have. Measured 2026-10-01: 5 x 262144 booted (KV 1,350,182 tokens /
+# 5.15x) but left ~0 bytes free at runtime, and a real client request
+# (995-token prompt, max_tokens=261149) killed the worker with
+# `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free`.
+# Dropping to four slots returns ~1.93 GiB per rank for activations/graphs.
+# At SEQS=3 bf16, 262144/245760 also OOMed at boot and 241664 was max
+# possible; at SEQS=6 fp8, six full-length contexts do NOT fit at all
+# (6 x 241664 asked 10.70 GiB against 9.18 GiB free).
 FRANK_PRIMARY_MAXLEN=262144
-FRANK_PRIMARY_SEQS=5
-FRANK_PRIMARY_KV_BYTES=10388766720
+FRANK_PRIMARY_SEQS=4
+FRANK_PRIMARY_KV_BYTES=8311013376
 FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=1024
 FRANK_PRIMARY_GATHER_CHUNKS=32

scripts/frank-primary.service

@@ -8,8 +8,10 @@
 #
 # Same port, same served alias; the harness models.yml id is unchanged.
 # The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container, so a crashed engine is a
-# nonzero ExecStart and Restart=on-failure is real.
+# `vllm serve`) and then waits on the container. NOTE: that wait can return
+# 0 even when the engine died mid-request (vLLM's API server drains and
+# exits clean on a worker fault), which is why the restart policy is
+# `always` — see the [Service] note.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
 # of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
@@ -23,6 +25,9 @@
 Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0
 
 [Service]
 User=dank
@@ -39,8 +44,20 @@ ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
 # Boot cost: FP8 payload page-in plus torch compile/graph capture —
 # ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
-Restart=on-failure
+# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
+# torch.OutOfMemoryError kills the engine while vLLM's API server still
+# exits 0, so ExecStart returned clean, systemd logged "Deactivated
+# successfully", and on-failure NEVER fired — the endpoint stayed down until
+# a human noticed. `always` restarts on any container death; an explicit
+# `systemctl stop` still wins (systemd does not restart a unit the operator
+# stopped), so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# Do not let the start-rate limiter give up after a crash loop.
+StartLimitIntervalSec=0
+# The engine pins tens of GiB of host RAM (expert host slabs); it is the
+# last process on this host that should be chosen by the kernel OOM killer.
+OOMScoreAdjust=-500
 
 [Install]
 WantedBy=multi-user.target