frank-large: host-RAM pin-gate preflight in serve-frank-large.sh

The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.

serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).

frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
This commit is contained in:
Frank
2026-10-06 12:58:34 -07:00
parent df6b5bb6e8
commit c71a3066c0
4 changed files with 86 additions and 0 deletions
@@ -143,6 +143,19 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
image digest in podman storage, byte-exact checkpoint + n-gram table +
vision tower, flat tokenizer with all six files as regular files,
GPU nodes present.
- **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
floor the n-gram lookup pages in from disk on every long prompt. A
RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
when the dying 27B engine freed RAM). The serve script now checks
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
(default 600 s) for the RAM to clear, so a transient dip boots once
instead of crash-looping; if it never clears it refuses loudly. The 16
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
separate, higher, overall-budget knob).
- Engine stdout goes to the log, NOT the journal. The FIRST request after
a (re)start can sit silent ~45–60 s on this engine class (health
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
+19
View File
@@ -20,6 +20,25 @@
ngram-mod still skipped (recurrent-state rollback law on the qwen35
hybrid). Runbook renamed + rewritten:
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`.
- **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
flash-server line, so the engine moved with the model. Abliterated base
(refusals gone), vision back (the v2 tower loads beside the checkpoint),
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
Runbook: `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
crash-looped 8 boots, each dying in ~5 s at the pin gate, and
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
again; `serve-frank-large.sh` now has a host-RAM preflight that checks
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
`FRANK_LARGE_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
it never does (the 16 GiB floor is the engine's, not
`HALOGEN_HOST_RESERVE_GIB`).
## 2026-10-05
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on