docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape

Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
This commit is contained in:
Frank
2026-10-13 19:28:25 -07:00
parent a6f55f4cc6
commit bae78e7dae
3 changed files with 218 additions and 48 deletions
+64 -31
View File
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 | | Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 22208 --vram-kv-mib 6912` = the 29,120 MiB/card ceiling), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 6,912 MiB ≈ 843k tokens — the only dtype that funds four slots, since bf16 at 4 × 200,000 costs 11,679 MiB of pool and would cut the slab to 17,441), **MTP spec decode depth 2**, `--max-num-seqs 4` (the capacity contract is 4 × 200,000; 4 × (1+2) = 12 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 4-slot fp8-KV shape 2026-10-13, the 2-slot bf16 shape (22208/6912, `--max-num-seqs 2`) is the documented rollback. | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770| | Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 | | Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
@@ -525,21 +525,26 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
Rollback: revert the swap commit — the fp8-native `.rad` Rollback: revert the swap commit — the fp8-native `.rad`
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk. (`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`, **Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`,
`--max-model-len 200000` (the harness contextWindow must quote this), `--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM `--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the two are ONE budget — replaced `--expert-vs-cache-ratio`; the two are ONE budget —
`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710 `weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype` consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`, **`fp8` — quantized KV, restored 2026-10-13 because it is the only dtype that
funds four slots at this window**, `--num-speculative-tokens 2`,
`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB `--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`, v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16 `--override-chat-template`). Boot ledger (2026-10-13): `kv cache 6.75 GiB …
tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's backs 4 sequences all at 200000 tokens at once, which is the worst case` —
386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool `kv_attn` page 26.00 KiB (fp8) at 843,276 tokens (the sizing law predicted
10,375 slots, cards settled 1.85 GiB free. 843,566), `kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB
× 8 slots (4 sequences × 2 copies) = 432 MiB; the sampler prints **12 rows**
(4 × (1+2), inside the 25-row `wht6` exactness bound); weights filled to
147 KiB of slack, cards settled ~2 GiB free, and the KV pool lends 5.87 GiB to
the expert slab (the fp8-era 3,543 MiB pool lent 2.69 GiB).
Shape notes carried from the fp8-native era and unchanged: wht6 gains, Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
@@ -551,35 +556,52 @@ on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth, measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246). accept 0.7246).
**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` — **The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
both halves. Measured per rank from the two boot ledgers: `kv_attn` (13 equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13 the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB dtype-independent terms that live in the same pool (~110 MiB a sequence of
of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` = recurrent state + 448 MiB of prefix checkpoints):
462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is `(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
paid twice on this engine, and half-refunded: idle KV VRAM is also the expert engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
the retired tcclaviger fork's torch.compile, not radiance) and it also drops where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
buffers + arena onto the weights side, so `weights + kv` is capped near `--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`. 25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
pair, never `--kv-cache-size-mib`.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert **The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it. `RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
204,800) refuses to start until both knobs are raised in the stop window by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
(sysfs write first, kernel cmdline + reboot only if read-only). before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
window is a convenience, not the repair; kernel cmdline persistence
(`ttm.pages_limit=52428800`) is the belt to it.
**Contract facts** (source-verified): the HTTP listener binds AFTER the **Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`, full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -761,6 +783,17 @@ linked runbooks.
the KV pool is worse than the OOM it prevents. the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check - Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom. `swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's - **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
@@ -253,6 +253,139 @@ from the scored configuration — bf16 can only reduce the attention-state
error, and a re-score is a drained-window job if the gate agreement is ever error, and a re-score is a drained-window job if the gate agreement is ever
needed in the record. needed in the record.
## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
### The outage: a refusal, not a crash
Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
after starting, with one journal line on repeat:
```text
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
```
**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
the name with a zero start time), so there was no engine, no port, and no engine
log — the signature of a preflight refusal, which every deployer/monitor view
outside the journal renders as an unexplained crash loop.
The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
runtime state, and its default is half of RAM** — so the manual raise of
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
script enforces. The fp8-native era had planned the persistent half of this
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
a power failure is not scheduled.
### The repair: the gate raises the knob itself
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
the MIN over the knobs the running kernel exposes, and when it is below
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
The unit is root and the knob is a 0644 root-owned module parameter, so the write
is legal; a kernel that refuses the move produces the same named refusal as
before, and a reboot is now a boot rather than a visit to the machine.
| Knob | Power-failure default | After the gate |
|---|---|---|
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
exposes* drive the gate against a temp knob directory — a below-floor writable
knob ends at the target and the start passes; a read-only knob leaves the cap
untouched and the start refuses by name; an absent `pages_max` is skipped, never
read as zero (a zero reading would refuse a healthy host).
### The shape: 4 slots × 200,000, fp8 KV
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
concurrent 200,000-token conversations with quantized KV.
fp8 is not a preference at four slots, it is the only dtype that affords them.
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
fourth slot is bought in KV, out of the one per-card budget
(`weights + kv ≤ 29,120 MiB`):
| dtype | pool for 4 × 200,000 | weights slab left |
|---|---|---|
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
1.93).
Terms that move with the dtype and the slot count:
- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
+110 MiB per slot, 440 MiB for four (already in the arithmetic above).
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
**12 rows**, still inside the 25-row bound at which the `wht6` wire is
bit-exact — so four slots cost nothing in wire exactness.
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
(200,000), which did not change; only the concurrency did.
Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
26,071 MiB slab and does not come back by quantizing KV.
### Boot ledger (2026-10-13 19:21, four slots, fp8)
The sizing law's prediction, checked against the engine's own arithmetic:
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
| Group | Page / state | Size | Tokens |
|---|---|---|---|
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
the worst case` — the capacity contract, in the engine's words. The rest of the
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
Two ledger lines that confirm the derived consequences: the sampler prints
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
(58.50 KiB, half of the bf16 entry).
### Recovery evidence
```text
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
204800 MiB (52428800 pages)
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
204800 MiB clears the pinned-ledger floor 150000
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
```
`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
the boot, i.e. the root `ExecStartPre` hook did the write the service process
(User=dank) cannot.
## Gates (measured at cutover) ## Gates (measured at cutover)
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`, - **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
+21 -17
View File
@@ -2,13 +2,13 @@
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon** **Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint, (digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL driving `podman run` directly, root systemd unit, :7770):
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision `ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for (`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`), TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the (explicit `--vram-weights-mib 22208 --vram-kv-mib 6912`), the 47.7 GiB PLE table
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3, pinned to host RAM, **fp8 KV at 4 × 200,000**, MTP spec decode depth 2,
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005 froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
and frank-archon-vllm-shapeG-20261003), and frank-archon-vllm-shapeG-20261003),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned (frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
@@ -36,10 +36,10 @@ graph LR
gfx1201, 32 GiB each] gfx1201, 32 GiB each]
B2[radiance StillDeadcode, digest-pinned image B2[radiance StillDeadcode, digest-pinned image
podman run, frank-archon, port 7770] podman run, frank-archon, port 7770]
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8 C[Swift1.5-Qwen3.8-Flash-Next GPTQ w4 .rad
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM, 2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3, PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
2 x 200000, froggeric v22.5] 4 x 200000, froggeric v22.5]
end end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -155,12 +155,16 @@ frank-zealot, port 7771]
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**: - [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container); 1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
the FP8-native payload remains the rollback. **2026-10-11: the KV cache is the FP8-native payload remains the rollback. **2026-10-13: 4 x 200,000 with
UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 → fp8 KV** — `--max-num-seqs 4` + `--kv-cache-dtype fp8` in the same 6912 MiB
462,752 tokens), weights slab 26071 → 22208, host pool 44928, pool (843,566 tokens), slab 22208 and host pool 44928 untouched. fp8 is what
`--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is funds the fourth slot: the pool must back seqs x window, and at bf16 four
dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%), slots cost 11,679 MiB, which would cut the resident slab to 17,441 (-33 %).
NOT decode, and spec-decode acceptance rose 0.661 → 0.740. The bf16 arm (2 x 200,000, 462,752 tokens, prefill -29 %, acceptance
0.661 → 0.740) is the documented rollback. **2026-10-13 also: the pinned-ledger
preflight now raises `ttm.pages_limit` itself** — the 2026-10-12 power failure
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the - [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)** w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine (root systemd unit, port 7770): the StillDeadcode **radiance** engine