docs(frank-archon): the slab is a slope, and the ceiling was mine not the engine's
Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 -> 3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %, wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s. Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at 29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable). The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the number, so the next resize starts from the engine's arithmetic. Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism, the pinned pool is the slow path. Capacity contract live-verified at 3 slots: 3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too large to ever fit. Probe law learned: the oversized-prompt refusal names the prompt's token count BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.
This commit is contained in:
Frank
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 22208 --vram-kv-mib 6912` = the 29,120 MiB/card ceiling), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 6,912 MiB ≈ 843k tokens — the only dtype that funds four slots, since bf16 at 4 × 200,000 costs 11,679 MiB of pool and would cut the slab to 17,441), **MTP spec decode depth 2**, `--max-num-seqs 4` (the capacity contract is 4 × 200,000; 4 × (1+2) = 12 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 4-slot fp8-KV shape 2026-10-13, the 2-slot bf16 shape (22208/6912, `--max-num-seqs 2`) is the documented rollback. | 7770 |
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
@@ -525,26 +525,35 @@ the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
|
||||
Rollback: revert the swap commit — the fp8-native `.rad`
|
||||
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
|
||||
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 4`,
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
|
||||
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
|
||||
partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
|
||||
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
|
||||
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
|
||||
`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
|
||||
consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
|
||||
**`fp8` — quantized KV, restored 2026-10-13 because it is the only dtype that
|
||||
funds four slots at this window**, `--num-speculative-tokens 2`,
|
||||
`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
|
||||
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
|
||||
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
|
||||
`--override-chat-template`). Boot ledger (2026-10-13): `kv cache 6.75 GiB …
|
||||
backs 4 sequences all at 200000 tokens at once, which is the worst case` —
|
||||
`kv_attn` page 26.00 KiB (fp8) at 843,276 tokens (the sizing law predicted
|
||||
843,566), `kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB
|
||||
× 8 slots (4 sequences × 2 copies) = 432 MiB; the sampler prints **12 rows**
|
||||
(4 × (1+2), inside the 25-row `wht6` exactness bound); weights filled to
|
||||
147 KiB of slack, cards settled ~2 GiB free, and the KV pool lends 5.87 GiB to
|
||||
the expert slab (the fp8-era 3,543 MiB pool lent 2.69 GiB).
|
||||
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
|
||||
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
|
||||
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
|
||||
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
|
||||
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
|
||||
because it is the only dtype that funds the slot count at this window**,
|
||||
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
|
||||
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
|
||||
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
|
||||
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
|
||||
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
|
||||
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
|
||||
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
|
||||
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
|
||||
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
|
||||
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
|
||||
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
|
||||
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
|
||||
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
|
||||
4.22 GiB to the expert cache. Prefill tracks this slab: the 24,000 → 25,600
|
||||
step measured +8.9 % single-stream (3,290 → 3,584 t/s at 43.5k) and +17.6 %
|
||||
concurrent (3-way 197k aggregate), so the pinned host pool is the slow path and
|
||||
every MiB moved from RAM onto the card is throughput.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
|
||||
@@ -194,12 +194,19 @@ FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
|
||||
# once` line is the referee for THIS pair and the concurrent probe is its
|
||||
# regression test.
|
||||
# FALSIFIABLE CLAIM for the 25,600 arm: today measured slab 22,208 -> 3,397
|
||||
# t/s and 24,000 -> 3,290 t/s at 43.5k single-stream, so below ~26 GiB the slab
|
||||
# is a THRESHOLD, not a slope. Expected: +1,600 resident expert slots, ~0
|
||||
# prefill. The point that DOES move prefill is 26,071 (the 2026-10-11 fp8 arm,
|
||||
# 4,379 t/s), fundable only by dropping to two slots. Prefill moving more than
|
||||
# +-3 % on this arm falsifies the threshold reading.
|
||||
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
|
||||
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
|
||||
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
|
||||
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
|
||||
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
|
||||
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
|
||||
# pair landed inside noise of EACH OTHER, which is what made it read as a
|
||||
# plateau; two points are not a curve.
|
||||
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
|
||||
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
|
||||
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
|
||||
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
|
||||
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
|
||||
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
|
||||
FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
|
||||
|
||||
@@ -145,11 +145,18 @@ dtype`) so a future resize cannot silently under-provision the pool.
|
||||
|
||||
### Why the weights slab pays — and why that is not pure loss
|
||||
|
||||
The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
|
||||
boot consumes outside both (driver context, the 990 MiB activation arena,
|
||||
hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
|
||||
**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
|
||||
resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
|
||||
The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
|
||||
hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
|
||||
24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
|
||||
context, the activation arena and --gpu-headroom-mib come out of it; anything
|
||||
past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
|
||||
against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
|
||||
static region)`. Claimable reserve above the ENGINE's floor:
|
||||
2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
|
||||
written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
|
||||
figure of 31,856 MiB; the engine never asked for that reserve and the seam now
|
||||
derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
|
||||
the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
|
||||
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
|
||||
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
|
||||
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
|
||||
@@ -303,7 +310,7 @@ knob ends at the target and the start passes; a read-only knob leaves the cap
|
||||
untouched and the start refuses by name; an absent `pages_max` is skipped, never
|
||||
read as zero (a zero reading would refuse a healthy host).
|
||||
|
||||
### The shape: 4 slots × 200,000, fp8 KV
|
||||
### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
|
||||
|
||||
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
|
||||
concurrent 200,000-token conversations with quantized KV.
|
||||
@@ -311,7 +318,7 @@ concurrent 200,000-token conversations with quantized KV.
|
||||
fp8 is not a preference at four slots, it is the only dtype that affords them.
|
||||
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
|
||||
fourth slot is bought in KV, out of the one per-card budget
|
||||
(`weights + kv ≤ 29,120 MiB`):
|
||||
(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
|
||||
|
||||
| dtype | pool for 4 × 200,000 | weights slab left |
|
||||
|---|---|---|
|
||||
@@ -345,7 +352,7 @@ cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
|
||||
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
|
||||
26,071 MiB slab and does not come back by quantizing KV.
|
||||
|
||||
### Boot ledger (2026-10-13 19:21, four slots, fp8)
|
||||
### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
|
||||
|
||||
The sizing law's prediction, checked against the engine's own arithmetic:
|
||||
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
|
||||
@@ -370,6 +377,50 @@ Two ledger lines that confirm the derived consequences: the sampler prints
|
||||
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
|
||||
(58.50 KiB, half of the bf16 entry).
|
||||
|
||||
### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
|
||||
|
||||
Frank's call, twice in one evening. Both are one-env-pair changes, and both were
|
||||
read off the boot ledger rather than assumed.
|
||||
|
||||
**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
|
||||
3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
|
||||
was backing 858,969 tokens — 43 % more window than the contract asks for.
|
||||
Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
|
||||
1,792 MiB to the slab and drops the all-reduce to 9 rows.
|
||||
|
||||
**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
|
||||
`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
|
||||
--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
|
||||
the ENGINE's own floor, with the budget table calling the surplus `throughput on
|
||||
the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
|
||||
`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
|
||||
|
||||
| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
|
||||
|---|---|---|---|---|
|
||||
| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
|
||||
| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
|
||||
| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
|
||||
| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
|
||||
|
||||
The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
|
||||
~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
|
||||
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
|
||||
slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
|
||||
plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
|
||||
the card is throughput (the ledger shows the transfer directly: slab
|
||||
+1,313 slots, pinned pool −1,313).
|
||||
|
||||
The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
|
||||
for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
|
||||
30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
|
||||
**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
|
||||
it is a capacity decision, not a tuning one.
|
||||
|
||||
Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
|
||||
of **199,984 tokens** each (599,952 tokens) were accepted and completed on
|
||||
`/v1/completions`; a fourth full-height request is refused as "too large to ever
|
||||
fit" — radiance never truncates.
|
||||
|
||||
### Recovery evidence
|
||||
|
||||
```text
|
||||
|
||||
@@ -6,8 +6,9 @@ driving `podman run` directly, root systemd unit, :7770):
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
|
||||
(`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
|
||||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 22208 --vram-kv-mib 6912`), the 47.7 GiB PLE table
|
||||
pinned to host RAM, **fp8 KV at 4 × 200,000**, MTP spec decode depth 2,
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
and frank-archon-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
@@ -155,14 +156,22 @@ frank-zealot, port 7771]
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-13: 4 x 200,000 with
|
||||
fp8 KV** — `--max-num-seqs 4` + `--kv-cache-dtype fp8` in the same 6912 MiB
|
||||
pool (843,566 tokens), slab 22208 and host pool 44928 untouched. fp8 is what
|
||||
funds the fourth slot: the pool must back seqs x window, and at bf16 four
|
||||
slots cost 11,679 MiB, which would cut the resident slab to 17,441 (-33 %).
|
||||
The bf16 arm (2 x 200,000, 462,752 tokens, prefill -29 %, acceptance
|
||||
0.661 → 0.740) is the documented rollback. **2026-10-13 also: the pinned-ledger
|
||||
preflight now raises `ttm.pages_limit` itself** — the 2026-10-12 power failure
|
||||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||||
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
|
||||
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
|
||||
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
|
||||
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
|
||||
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
|
||||
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
|
||||
prediction that the slab was a threshold below 26 GiB died on its own
|
||||
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
|
||||
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
|
||||
and the 4-slot fp8 shape are the documented rollbacks.
|
||||
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
|
||||
itself** — the 2026-10-12 power failure
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
|
||||
@@ -32,6 +32,41 @@
|
||||
aggregate prefill, no preemption; prefix cache verified (repeat answered in
|
||||
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
|
||||
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
|
||||
- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
|
||||
4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
|
||||
was funding 858,969 tokens — 43 % over the contract. Then slab
|
||||
24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
|
||||
OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||||
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
|
||||
real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
|
||||
invented on 2026-10-11. The pair now spends 30,720 and settles
|
||||
`unclaimed 477.30 MiB` (cards ~336 MiB free).
|
||||
- **The prediction shipped with that arm was falsified by its own measurement,
|
||||
which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
|
||||
below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
|
||||
other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
|
||||
cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
|
||||
193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
|
||||
pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
|
||||
points inside noise of each other are not a plateau. The cliff is above
|
||||
25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
|
||||
slots (26,112 + 5,120 > the ceiling) and free at two.
|
||||
- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
|
||||
accepted and completed (599,952 tokens resident), ledger
|
||||
`backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
|
||||
law's 608,027. A fourth full-height request is refused as too large to ever
|
||||
fit — radiance never truncates. Probe law: the oversized-prompt counter is
|
||||
FREE (the refusal fires when PROMPT tokens alone exceed the window —
|
||||
`prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
|
||||
window is ~514,286 characters at the measured 2.5714 chars/token, so a
|
||||
calibration probe below that pays a real prefill instead of reporting.
|
||||
- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
|
||||
exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
|
||||
conversation. Untested arm; capacity decision, not tuning.
|
||||
- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
|
||||
owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
|
||||
measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
|
||||
own test. 371/371 tests and e2e 33/33 on both deploys.
|
||||
- **Power-failure casualties, NOT fixed (outside the archon window):**
|
||||
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
|
||||
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
|
||||
|
||||
Reference in New Issue
Block a user