docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape
Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law, the shape arithmetic, the boot ledger that confirms it: 843,276 tokens = 4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise lesson under Key Lessons, and the pinned-RAM paragraph now points at scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
This commit is contained in:
Frank
@@ -253,6 +253,139 @@ from the scored configuration — bf16 can only reduce the attention-state
|
||||
error, and a re-score is a drained-window job if the gate agreement is ever
|
||||
needed in the record.
|
||||
|
||||
## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
|
||||
|
||||
### The outage: a refusal, not a crash
|
||||
|
||||
Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
|
||||
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
|
||||
after starting, with one journal line on repeat:
|
||||
|
||||
```text
|
||||
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
|
||||
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
|
||||
```
|
||||
|
||||
**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
|
||||
the name with a zero start time), so there was no engine, no port, and no engine
|
||||
log — the signature of a preflight refusal, which every deployer/monitor view
|
||||
outside the journal renders as an unexplained crash loop.
|
||||
|
||||
The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
|
||||
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
|
||||
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
|
||||
runtime state, and its default is half of RAM** — so the manual raise of
|
||||
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
|
||||
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
|
||||
script enforces. The fp8-native era had planned the persistent half of this
|
||||
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
|
||||
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
|
||||
a power failure is not scheduled.
|
||||
|
||||
### The repair: the gate raises the knob itself
|
||||
|
||||
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
|
||||
the MIN over the knobs the running kernel exposes, and when it is below
|
||||
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
|
||||
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
|
||||
The unit is root and the knob is a 0644 root-owned module parameter, so the write
|
||||
is legal; a kernel that refuses the move produces the same named refusal as
|
||||
before, and a reboot is now a boot rather than a visit to the machine.
|
||||
|
||||
| Knob | Power-failure default | After the gate |
|
||||
|---|---|---|
|
||||
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
|
||||
|
||||
Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
|
||||
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
|
||||
exposes* drive the gate against a temp knob directory — a below-floor writable
|
||||
knob ends at the target and the start passes; a read-only knob leaves the cap
|
||||
untouched and the start refuses by name; an absent `pages_max` is skipped, never
|
||||
read as zero (a zero reading would refuse a healthy host).
|
||||
|
||||
### The shape: 4 slots × 200,000, fp8 KV
|
||||
|
||||
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
|
||||
concurrent 200,000-token conversations with quantized KV.
|
||||
|
||||
fp8 is not a preference at four slots, it is the only dtype that affords them.
|
||||
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
|
||||
fourth slot is bought in KV, out of the one per-card budget
|
||||
(`weights + kv ≤ 29,120 MiB`):
|
||||
|
||||
| dtype | pool for 4 × 200,000 | weights slab left |
|
||||
|---|---|---|
|
||||
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
|
||||
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
|
||||
|
||||
So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
|
||||
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
|
||||
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
|
||||
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
|
||||
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
|
||||
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
|
||||
1.93).
|
||||
|
||||
Terms that move with the dtype and the slot count:
|
||||
|
||||
- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
|
||||
+110 MiB per slot, 440 MiB for four (already in the arithmetic above).
|
||||
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
|
||||
**12 rows**, still inside the 25-row bound at which the `wht6` wire is
|
||||
bit-exact — so four slots cost nothing in wire exactness.
|
||||
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
|
||||
58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
|
||||
bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
|
||||
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
|
||||
(200,000), which did not change; only the concurrency did.
|
||||
|
||||
Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
|
||||
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
|
||||
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
|
||||
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
|
||||
26,071 MiB slab and does not come back by quantizing KV.
|
||||
|
||||
### Boot ledger (2026-10-13 19:21, four slots, fp8)
|
||||
|
||||
The sizing law's prediction, checked against the engine's own arithmetic:
|
||||
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
|
||||
|
||||
| Group | Page / state | Size | Tokens |
|
||||
|---|---|---|---|
|
||||
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
|
||||
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
|
||||
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
|
||||
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
|
||||
|
||||
`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
|
||||
the worst case` — the capacity contract, in the engine's words. The rest of the
|
||||
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
|
||||
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
|
||||
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
|
||||
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
|
||||
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
|
||||
|
||||
Two ledger lines that confirm the derived consequences: the sampler prints
|
||||
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
|
||||
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
|
||||
(58.50 KiB, half of the bf16 entry).
|
||||
|
||||
### Recovery evidence
|
||||
|
||||
```text
|
||||
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
|
||||
is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
|
||||
204800 MiB (52428800 pages)
|
||||
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
|
||||
204800 MiB clears the pinned-ledger floor 150000
|
||||
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
|
||||
```
|
||||
|
||||
`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
|
||||
the boot, i.e. the root `ExecStartPre` hook did the write the service process
|
||||
(User=dank) cannot.
|
||||
|
||||
|
||||
## Gates (measured at cutover)
|
||||
|
||||
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
driving `podman run` directly, root systemd unit, :7770):
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
|
||||
(`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
|
||||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 22208 --vram-kv-mib 6912`), the 47.7 GiB PLE table
|
||||
pinned to host RAM, **fp8 KV at 4 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
and frank-archon-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
@@ -36,10 +36,10 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-archon, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
2 x 200000, froggeric v22.5]
|
||||
C[Swift1.5-Qwen3.8-Flash-Next GPTQ w4 .rad
|
||||
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
|
||||
4 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -155,12 +155,16 @@ frank-zealot, port 7771]
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
|
||||
UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
|
||||
462,752 tokens), weights slab 26071 → 22208, host pool 44928,
|
||||
`--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
|
||||
dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
|
||||
NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
|
||||
the FP8-native payload remains the rollback. **2026-10-13: 4 x 200,000 with
|
||||
fp8 KV** — `--max-num-seqs 4` + `--kv-cache-dtype fp8` in the same 6912 MiB
|
||||
pool (843,566 tokens), slab 22208 and host pool 44928 untouched. fp8 is what
|
||||
funds the fourth slot: the pool must back seqs x window, and at bf16 four
|
||||
slots cost 11,679 MiB, which would cut the resident slab to 17,441 (-33 %).
|
||||
The bf16 arm (2 x 200,000, 462,752 tokens, prefill -29 %, acceptance
|
||||
0.661 → 0.740) is the documented rollback. **2026-10-13 also: the pinned-ledger
|
||||
preflight now raises `ttm.pages_limit` itself** — the 2026-10-12 power failure
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
|
||||
Reference in New Issue
Block a user