dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: host-RAM pin-gate preflight in serve-frank-large.sh

Browse Source 
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.

serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).

frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
This commit is contained in:
Frank
2026-10-06 12:58:34 -07:00
parent df6b5bb6e8
commit c71a3066c0
4 changed files with 86 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md
+13
View File
@@ -143,6 +143,19 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
image digest in podman storage, byte-exact checkpoint + n-gram table +
vision tower, flat tokenizer with all six files as regular files,
GPU nodes present.
- **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
floor the n-gram lookup pages in from disk on every long prompt. A
RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
when the dying 27B engine freed RAM). The serve script now checks
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
(default 600 s) for the RAM to clear, so a transient dip boots once
instead of crash-looping; if it never clears it refuses loudly. The 16
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
separate, higher, overall-budget knob).
- Engine stdout goes to the log, NOT the journal. The FIRST request after
a (re)start can sit silent ~45–60 s on this engine class (health
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.