frank-large: host-RAM pin-gate preflight in serve-frank-large.sh
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42): the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB, 3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor). It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the host with only ~1.8 GiB MemAvailable - a restart would crash-loop again. serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM to clear, so a transient dip boots once instead of crash-looping; if it never clears it refuses loudly. The 16 GiB floor is the engine's (not HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob). frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook + log: the pin-gate law, the incident, and the fix.
This commit is contained in:
@@ -143,6 +143,19 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
image digest in podman storage, byte-exact checkpoint + n-gram table +
|
||||
vision tower, flat tokenizer with all six files as regular files,
|
||||
GPU nodes present.
|
||||
- **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
|
||||
refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
|
||||
plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
|
||||
floor the n-gram lookup pages in from disk on every long prompt. A
|
||||
RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
|
||||
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
|
||||
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
|
||||
when the dying 27B engine freed RAM). The serve script now checks
|
||||
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
|
||||
(default 600 s) for the RAM to clear, so a transient dip boots once
|
||||
instead of crash-looping; if it never clears it refuses loudly. The 16
|
||||
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
|
||||
separate, higher, overall-budget knob).
|
||||
- Engine stdout goes to the log, NOT the journal. The FIRST request after
|
||||
a (re)start can sit silent ~45–60 s on this engine class (health
|
||||
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
|
||||
|
||||
@@ -20,6 +20,25 @@
|
||||
ngram-mod still skipped (recurrent-state rollback law on the qwen35
|
||||
hybrid). Runbook renamed + rewritten:
|
||||
`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`.
|
||||
- **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server
|
||||
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
|
||||
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
|
||||
flash-server line, so the engine moved with the model. Abliterated base
|
||||
(refusals gone), vision back (the v2 tower loads beside the checkpoint),
|
||||
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
|
||||
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
|
||||
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
|
||||
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
|
||||
crash-looped 8 boots, each dying in ~5 s at the pin gate, and
|
||||
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
|
||||
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
|
||||
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
|
||||
again; `serve-frank-large.sh` now has a host-RAM preflight that checks
|
||||
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
|
||||
`FRANK_LARGE_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
|
||||
it never does (the 16 GiB floor is the engine's, not
|
||||
`HALOGEN_HOST_RESERVE_GIB`).
|
||||
|
||||
## 2026-10-05
|
||||
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
|
||||
|
||||
Reference in New Issue
Block a user