docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape

The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the
--vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and
bats carried the new shape while the page and the AGENTS.md block still
described the 1.0.13 3-slot era, so every point-of-use number was wrong.

Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not
from the stale pre-#31 draft: runtime block (digest, version oracle, the two
CLI breaks that are boot refusals), shape table (demand-signal rows, derived
VRAM split, restored spec/batch row, host-only prefix tier, step sharing),
container contract (no /data mount), the measured 2-slot ledger, a new Eras row
for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and
the laws: pool sizing and the per-card budget re-scoped to the pair era, a new
resolver law (funds ~90% of the worst case, host RAM below a slab step is inert,
MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the
fix ordered-not-verified, the knob-availability law flipped, and the slots law
now pricing the third slot in KV (4.20 -> 2.76 GiB).
This commit is contained in:
Frank
2026-10-19 14:55:43 -07:00
parent 3d10a4ca4f
commit d9b82b372e
2 changed files with 112 additions and 63 deletions
+9 -6
View File
@@ -462,12 +462,15 @@ disagrees with it is the bug.
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`, - Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned `scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name. `stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`, - Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair `expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤ MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
+ 64 GiB disk, n-gram table pinned to RAM. by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk
tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal,
not a no-op.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000 - Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger (over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
is why this is a root unit, and `ttm-preflight.sh` raises is why this is a root unit, and `ttm-preflight.sh` raises
+103 -57
View File
@@ -1,6 +1,6 @@
--- ---
type: reference type: reference
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)" title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.8.1, publisher Qwen3.8-Flash-Next fp8-iq4r-moe .rad, 2× R9700 TP=2 (ACTIVE)"
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught." description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
tags: tags:
- llm - llm
@@ -12,7 +12,7 @@ tags:
- tp2 - tp2
- expert-offload - expert-offload
- endpoint - endpoint
timestamp: 2026-10-18 timestamp: 2026-10-19
--- ---
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770 # frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
@@ -20,8 +20,9 @@ timestamp: 2026-10-18
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
path) driving `podman run` straight from the digest-pinned image under a **root** systemd path) driving `podman run` straight from the digest-pinned image under a **root** systemd
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
is a `MODEL_FILE` line plus a re-download — see Eras. is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws).
A payload swap is a `MODEL_FILE` line plus a re-download — see Eras.
## Current shape ## Current shape
@@ -44,14 +45,20 @@ declared size + sha256) as the transient unit `iq4r-fetch` (log
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only `/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
`.rad` files are mounted. `.rad` files are mounted.
**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e` **Runtime** — `stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed`
= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's = pin **1.8.1** (adopted 2026-10-18). The version oracle is the boot's own `STARTUP REPORT --
**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest, radiance 1.8.1` line, because `podman inspect` reports only the digest. The image lives in dank's
never the image ID (`69d05724…` is the config digest and sends `podman run` to the network). **rootless** storage and the host needs nothing but the amdgpu driver. Pin the INDEX manifest
The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one digest, never the image ID (`69d05724…` is the config digest and sends `podman run` to the
command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer network). The serve script's gate is that digest ref itself — `podman image exists name@sha256:…`
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the in one command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry. `latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws). **Two CLI breaks ride this pin,
and both are boot refusals, never silently-ignored flags:** the `--vram-weights-mib`/`--vram-kv-mib`
pair died at 1.3.0 (the resolver derives the split) and `--prefix-cache-dir` /
`--prefix-cache-disk-mib` died with the disk tier. A rollback to 1.0.13 (`sha256:4c3f4845…`) is
therefore a trio rewrite, not a digest swap — the Eras row carries it, and
`tests/frank_archon_unit.bats` asserts the retired knobs are ABSENT from env and argv so a stale
line cannot reach a boot.
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header) **Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image `scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
@@ -64,8 +71,10 @@ the name registered, and every one of those identities holds this GPU. Container
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the (publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring — cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune` memlock=-1:-1`, `/models` ro + the rad-tune cache at
template as a single-file mount, then `exec podman wait`: the script's lifetime IS the `/opt/radiance/share/radiance/tune` + the template as a single-file mount (there is **no `/data`**
any more: it left with the prefix disk tier, and `FRANK_ARCHON_STATE_DIR` is now only the host-side
directory that holds `tune/`), then `exec podman wait`: the script's lifetime IS the
container's, so a crash is a nonzero `ExecStart` and the restart policy is real. container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank` Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the `LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
@@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value | | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---| |---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default | | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) | | Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) | | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` | | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** | | Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) | | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | — | | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens** Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision `vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
live (≤ 4,194,304 px, 16,384 patches/pass). **26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
(≤ 4,194,304 px, 16,384 patches/pass).
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin, anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32 per-boot VRAM planner to second-guess. For comparison, the publisher's own suggested run line on 32
GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the GB cards was `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3 — a
1.0.x-era line, and `--expert-vs-cache-ratio` was removed at 1.2.0, so today it is a capacity floor
quoted from the card, not a run line this pin accepts; the
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
one string). one string).
@@ -130,6 +145,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) | | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) | | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account | | 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
@@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry 2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer` tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16. lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype- - **Pool sizing law (the explicit-pair era — at 1.8.1 the cache is derived, so the resolver law below
owns it).** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/ independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 × tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 − 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
@@ -174,13 +191,15 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21 pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is its regression test. A fourth full-height conversation needs 6,601 MiB of pool: refusing it is
Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change. Frank's explicit call, and under that pin serving it was a `--vram-kv-mib` change, not a code change.
At 1.8.1 the same decision is `MAX_SEQS` — the resolver buys the cache the demand signal implies.
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a - **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size. `seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget - **One VRAM budget per card** (the arithmetic below is the pair era's; the ceiling survives, the knobs
do not): `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and "*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with `--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
@@ -188,12 +207,34 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849 static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120 = 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the ("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective). buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
hand. Three boots of evidence:
| shape | demand | cache given | backing |
|---|---|---|---|
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today) 256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 % on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
@@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused). GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine - **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot): 3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
@@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9. [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only - **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
steps after each chunk) are absent from 1.0.13: proven on the running container shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
re-scores draft acceptance before it is trusted. edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it it is trusted.
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs × - **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06: declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2 (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
specialization, and radiance booted seqs 2 on its first fp8-native start. floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
**Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM — the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
"doesn't fit"; the container-side procedure and the kernel gate are . fp8-native start.
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
"does not fit" without that line has not read the runtime.
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only - **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable. thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)` - **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`