frank-primary: four full windows + Restart=always (runtime survivability)

Five 262144-token contexts passed every boot gate (KV 1,350,182 / 5.15x) but
left 0 bytes free at runtime: a real client request (995-token prompt,
max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1
with torch.OutOfMemoryError 'Tried to allocate 50.00 MiB ... 0 bytes free'.
Boot is not proof of runtime survivability.

Four slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per
rank for activations/graph capture.

Unit: Restart=always instead of on-failure — a runtime worker fault makes
vLLM's API server exit clean, so ExecStart returned 0 and on-failure never
fired, leaving the endpoint down until a human noticed. Also
StartLimitIntervalSec=0 and OOMScoreAdjust=-500.

scripts/frank-primary.service

@@ -8,8 +8,10 @@
 #
 # Same port, same served alias; the harness models.yml id is unchanged.
 # The serve script drives `podman run` directly (the image entrypoint execs
-# `vllm serve`) and then waits on the container, so a crashed engine is a
-# nonzero ExecStart and Restart=on-failure is real.
+# `vllm serve`) and then waits on the container. NOTE: that wait can return
+# 0 even when the engine died mid-request (vLLM's API server drains and
+# exits clean on a worker fault), which is why the restart policy is
+# `always` — see the [Service] note.
 #
 # WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
 # of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
@@ -23,6 +25,9 @@
 Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
 After=network-online.target
 Wants=network-online.target
+# Never let the start-rate limiter give up after a crash loop (a failed LLM
+# endpoint that systemd refuses to retry is worse than a retrying one).
+StartLimitIntervalSec=0
 
 [Service]
 User=dank
@@ -39,8 +44,20 @@ ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
 # Boot cost: FP8 payload page-in plus torch compile/graph capture —
 # ~15 min cold (measured 2026-09-29), faster warm.
 TimeoutStartSec=1800
-Restart=on-failure
+# Restart=always, NOT on-failure (measured 2026-10-01): a runtime
+# torch.OutOfMemoryError kills the engine while vLLM's API server still
+# exits 0, so ExecStart returned clean, systemd logged "Deactivated
+# successfully", and on-failure NEVER fired — the endpoint stayed down until
+# a human noticed. `always` restarts on any container death; an explicit
+# `systemctl stop` still wins (systemd does not restart a unit the operator
+# stopped), so bench windows and deploys are unaffected.
+Restart=always
 RestartSec=10
+# Do not let the start-rate limiter give up after a crash loop.
+StartLimitIntervalSec=0
+# The engine pins tens of GiB of host RAM (expert host slabs); it is the
+# last process on this host that should be chosen by the kernel OOM killer.
+OOMScoreAdjust=-500
 
 [Install]
 WantedBy=multi-user.target