dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape

Browse Source 
Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
This commit is contained in:
Frank
2026-10-13 19:28:25 -07:00
parent a6f55f4cc6
commit bae78e7dae
3 changed files with 218 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+64 -31
View File
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 22208 --vram-kv-mib 6912` = the 29,120 MiB/card ceiling), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 6,912 MiB ≈ 843k tokens — the only dtype that funds four slots, since bf16 at 4 × 200,000 costs 11,679 MiB of pool and would cut the slab to 17,441), **MTP spec decode depth 2**, `--max-num-seqs 4` (the capacity contract is 4 × 200,000; 4 × (1+2) = 12 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 4-slot fp8-KV shape 2026-10-13, the 2-slot bf16 shape (22208/6912, `--max-num-seqs 2`) is the documented rollback. | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
@@ -525,21 +525,26 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
Rollback: revert the swap commit — the fp8-native `.rad`
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`,
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`,
**`fp8` — quantized KV, restored 2026-10-13 because it is the only dtype that
funds four slots at this window**, `--num-speculative-tokens 2`,
`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16
tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's
386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool
10,375 slots, cards settled 1.85 GiB free.
`--override-chat-template`). Boot ledger (2026-10-13): `kv cache 6.75 GiB …
backs 4 sequences all at 200000 tokens at once, which is the worst case` —
`kv_attn` page 26.00 KiB (fp8) at 843,276 tokens (the sizing law predicted
843,566), `kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB
× 8 slots (4 sequences × 2 copies) = 432 MiB; the sampler prints **12 rows**
(4 × (1+2), inside the 25-row `wht6` exactness bound); weights filled to
147 KiB of slack, cards settled ~2 GiB free, and the KV pool lends 5.87 GiB to
the expert slab (the fp8-era 3,543 MiB pool lent 2.69 GiB).
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
@@ -551,35 +556,52 @@ on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` —
one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what
frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on
both halves. Measured per rank from the two boot ledgers: `kv_attn` (13
layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13
layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not
attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at
bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms
that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB
of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` =
462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is
paid twice on this engine, and half-refunded: idle KV VRAM is also the expert
cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the
fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots
× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance) and it also drops
the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is
bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
buffers + arena onto the weights side, so `weights + kv` is capped near
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
dtype-independent terms that live in the same pool (~110 MiB a sequence of
recurrent state + 448 MiB of prefix checkpoints):
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
pair, never `--kv-cache-size-mib`.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host
204,800) refuses to start until both knobs are raised in the stop window
(sysfs write first, kernel cmdline + reboot only if read-only).
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
window is a convenience, not the repair; kernel cmdline persistence
(`ttm.pages_limit=52428800`) is the belt to it.
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -761,6 +783,17 @@ linked runbooks.
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed