The 2026-10-29 incident, and the four gates that make it unrepeatable.
What happened
The rig's calibration boot died (startup stopped: out of memory — the BF16 n-gram table pins
95.37 GiB beside a 65.59 GiB/rank pool: 230.22 GiB pinned of 246.07, 8.06 GiB available). The
chain's health loop expired without failing, and the pusher then fed four rounds of 10M tokens
— 86 minutes of live prefill — into restored production, recording nothing.
The changes
File
Change
scripts/run-qwen38-bf16.sh
--ngram-placement is a knob: calib defaults to disk, kldref/score keep ram (their numbers are only comparable at an identical shape). calib writes .rig-boot (container, image, pid) into the tap dir and retires it on a non-zero exit.
scripts/calib-push.py
Refuses before the first prompt unless the marker exists, names this container, and that container is running.
scripts/qwen38-calib-window.sh
The gated calibration window: stop on a failed phase, never fall through the health wait; bank the 336 GiB reference container into trash only when both phases exit 0 and the tap holds grams; --dry-run prints the whole plan.
scripts/kld-gate-qwen38.sh
Two-report mode (candidate vs the served payload) for the disk where the 172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
serverAppConfigs-docs/LLM/frank-archon.md
The pipeline laws, and a Files section that names the live campaign instead of the retired swift15 era.
Evidence
./tests/run.sh → 587 ok / 0 fail, including 13 new behavioral cases: the dry-run argv's
n-gram tier per mode, an unknown tier refused by name, the pusher's three marker refusals
(no marker / wrong container / container not running, with a stubbed podman), the two-report
gate passing a genuine win, failing a better-mean-worse-tail report, and failing a win that
regresses one source slice, plus the window script's refusal and its gated dry-run plan.
shellcheck -S warning clean on the three shell scripts; py_compile clean.
Next after this lands: the overnight calibration drain (scripts/qwen38-calib-window.sh on
dankserver), then grams → imatrix → trash the reference container → one cd3 convert → score →
two-report gate → deploy if it beats the served payload.
The 2026-10-29 incident, and the four gates that make it unrepeatable.
## What happened
The rig's calibration boot died (`startup stopped: out of memory` — the BF16 n-gram table pins
95.37 GiB beside a 65.59 GiB/rank pool: 230.22 GiB pinned of 246.07, 8.06 GiB available). The
chain's health loop expired **without failing**, and the pusher then fed four rounds of 10M tokens
— 86 minutes of live prefill — into **restored production**, recording nothing.
## The changes
| File | Change |
|---|---|
| `scripts/run-qwen38-bf16.sh` | `--ngram-placement` is a knob: calib defaults to `disk`, kldref/score keep `ram` (their numbers are only comparable at an identical shape). calib writes `.rig-boot` (container, image, pid) into the tap dir and retires it on a non-zero exit. |
| `scripts/calib-push.py` | Refuses before the first prompt unless the marker exists, names *this* container, and that container is running. |
| `scripts/qwen38-calib-window.sh` | The gated calibration window: stop on a failed phase, never fall through the health wait; bank the 336 GiB reference container into trash only when both phases exit 0 and the tap holds grams; `--dry-run` prints the whole plan. |
| `scripts/kld-gate-qwen38.sh` | Two-report mode (candidate vs the served payload) for the disk where the 172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never read: a **p99.9 tail ceiling** and **every `by_source` slice** passing mean, tail and top-1. |
| `serverAppConfigs-docs/LLM/frank-archon.md` | The pipeline laws, and a Files section that names the live campaign instead of the retired swift15 era. |
## Evidence
`./tests/run.sh` → **587 ok / 0 fail**, including 13 new behavioral cases: the dry-run argv's
n-gram tier per mode, an unknown tier refused by name, the pusher's three marker refusals
(no marker / wrong container / container not running, with a stubbed `podman`), the two-report
gate passing a genuine win, failing a better-mean-worse-tail report, and failing a win that
regresses one source slice, plus the window script's refusal and its gated dry-run plan.
`shellcheck -S warning` clean on the three shell scripts; `py_compile` clean.
Next after this lands: the overnight calibration drain (`scripts/qwen38-calib-window.sh` on
dankserver), then grams → imatrix → trash the reference container → one `cd3` convert → score →
two-report gate → deploy if it beats the served payload.
---
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98
Four gates, all bought with the 2026-10-29 incident: the rig's calibration boot died of an OOM,
the chain's health loop expired without failing, and the pusher fed four rounds of 10M tokens
(86 minutes of prefill) into RESTORED PRODUCTION, recording nothing.
- run-qwen38-bf16.sh: --ngram-placement becomes a knob whose calib default is `disk` (the 95.37
GiB table pin is what starved that boot: 230.22 GiB pinned of 246, 8.06 GiB available) while
kldref and score keep `ram`. calib writes .rig-boot into the tap dir and retires it on a
non-zero exit, so a dead boot cannot leave the tap looking rig-authorized.
- calib-push.py: refuses before the first prompt unless the marker exists, names this container,
and that container is running. With the driver's existing tap-progress gate, a push can no
longer spend hours feeding an endpoint that is not the rig.
- qwen38-calib-window.sh: the calibration window as a gated chain — stop on a failed phase
instead of falling through the health wait, bank the 336 GiB reference container into trash
only when both phases exit 0 and the tap holds grams, and print the whole plan on --dry-run.
- kld-gate-qwen38.sh: two-report mode (candidate vs the served payload) for the disk where the
172.8 GiB FP8 anchor cannot live, plus the laws the report always carried and the gate never
read: a p99.9 tail ceiling and every by_source slice passing mean, tail and top-1.
- frank-archon.md: the pipeline laws (marker, n-gram tier, three-metric acceptance) and a Files
section that names the live campaign instead of the retired swift15 era.
Tests: 587/0 — 13 new behavioral cases (dry-run argv, marker refusals, two-report gate math).
dank
merged commit 92a412e99d into main2026-10-29 16:14:35 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The 2026-10-29 incident, and the four gates that make it unrepeatable.
What happened
The rig's calibration boot died (
startup stopped: out of memory— the BF16 n-gram table pins95.37 GiB beside a 65.59 GiB/rank pool: 230.22 GiB pinned of 246.07, 8.06 GiB available). The
chain's health loop expired without failing, and the pusher then fed four rounds of 10M tokens
— 86 minutes of live prefill — into restored production, recording nothing.
The changes
scripts/run-qwen38-bf16.sh--ngram-placementis a knob: calib defaults todisk, kldref/score keepram(their numbers are only comparable at an identical shape). calib writes.rig-boot(container, image, pid) into the tap dir and retires it on a non-zero exit.scripts/calib-push.pyscripts/qwen38-calib-window.sh--dry-runprints the whole plan.scripts/kld-gate-qwen38.shby_sourceslice passing mean, tail and top-1.serverAppConfigs-docs/LLM/frank-archon.mdEvidence
./tests/run.sh→ 587 ok / 0 fail, including 13 new behavioral cases: the dry-run argv'sn-gram tier per mode, an unknown tier refused by name, the pusher's three marker refusals
(no marker / wrong container / container not running, with a stubbed
podman), the two-reportgate passing a genuine win, failing a better-mean-worse-tail report, and failing a win that
regresses one source slice, plus the window script's refusal and its gated dry-run plan.
shellcheck -S warningclean on the three shell scripts;py_compileclean.Next after this lands: the overnight calibration drain (
scripts/qwen38-calib-window.shondankserver), then grams → imatrix → trash the reference container → one
cd3convert → score →two-report gate → deploy if it beats the served payload.
cwd: /home/blank/.omp/wt/wt-20261027-142335-6d1fc1f · session: 01a17f3f-3e2e-7509-aef8-be516c27cf98