Merge pull request 'docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape' (#38) from docs/frank-archon-181-two-slot into main
This commit was merged in pull request #38.
This commit is contained in:
@@ -462,12 +462,15 @@ disagrees with it is the bug.
|
||||
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
|
||||
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
|
||||
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
|
||||
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
|
||||
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
|
||||
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
|
||||
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
|
||||
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
|
||||
+ 64 GiB disk, n-gram table pinned to RAM.
|
||||
`stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
|
||||
- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
|
||||
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
|
||||
MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
|
||||
tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
|
||||
by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
|
||||
the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk
|
||||
tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal,
|
||||
not a no-op.
|
||||
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
|
||||
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
|
||||
is why this is a root unit, and `ttm-preflight.sh` raises
|
||||
|
||||
@@ -251,21 +251,20 @@ Knobs, in the order worth trying them (radiance GUIDE §4.5):
|
||||
the prefill stager's smallest pass — the startup line names it, and below it every
|
||||
routed layer reads its pooled experts synchronously, which costs the prefill far
|
||||
more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
|
||||
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The
|
||||
vendor's curve on a 48-layer MoE at 200k on two cards: chunk 512 → 1750 t/s, 1024
|
||||
→ 2200, 2048 → 2706 (a 185k prompt: 138 s against 91 s), so halving the stall
|
||||
costs ~19 % of prefill. Raising it past the interval buys nothing and loses expert
|
||||
residency to the arena.
|
||||
- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step
|
||||
while every shared step is planned at a smaller shared budget, which on an
|
||||
expert-tiered box lends that arena back to the expert slab. Vendor TP2
|
||||
expert-tiered TTFT deltas vs a plain 2048 budget: −6.1 % at 2k, +2.3 % at 18k,
|
||||
+49.1 % at 250k — long prompts only.
|
||||
- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own
|
||||
curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE
|
||||
at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of
|
||||
magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses
|
||||
expert residency to the arena.
|
||||
- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every
|
||||
shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena
|
||||
back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing
|
||||
at short prompts, a large TTFT gain at the far end of the window.
|
||||
|
||||
Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
|
||||
expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would
|
||||
have had (320 MiB/card at TP2 for a 4096 budget), so the decode rows inside that step
|
||||
also read experts over the link — a smaller shared budget helps twice. And **re-check
|
||||
expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the
|
||||
decode rows inside that step also read experts over the link — a smaller shared budget helps twice.
|
||||
And **re-check
|
||||
`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
|
||||
pool are the same memory, and the failure looks like turn two taking as long as turn
|
||||
one.
|
||||
@@ -276,8 +275,9 @@ Availability is a version question, not a config question: `--busy-prefill-chunk
|
||||
tree, and check the source clone's depth before dating a flag from `git log -S`
|
||||
(a `--depth 1` clone reports its own tip for every search).
|
||||
|
||||
Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = interval
|
||||
2048) — the measured census and the verdict for this fleet's shape are that page's.
|
||||
Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 =
|
||||
interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict
|
||||
for this fleet's shape are that page's.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)"
|
||||
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.8.1, publisher Qwen3.8-Flash-Next fp8-iq4r-moe .rad, 2× R9700 TP=2 (ACTIVE)"
|
||||
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
|
||||
tags:
|
||||
- llm
|
||||
@@ -12,7 +12,7 @@ tags:
|
||||
- tp2
|
||||
- expert-offload
|
||||
- endpoint
|
||||
timestamp: 2026-10-18
|
||||
timestamp: 2026-10-19
|
||||
---
|
||||
|
||||
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
|
||||
@@ -20,8 +20,9 @@ timestamp: 2026-10-18
|
||||
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
|
||||
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
|
||||
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
|
||||
PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap
|
||||
is a `MODEL_FILE` line plus a re-download — see Eras.
|
||||
PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
|
||||
is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws).
|
||||
A payload swap is a `MODEL_FILE` line plus a re-download — see Eras.
|
||||
|
||||
## Current shape
|
||||
|
||||
@@ -44,14 +45,20 @@ declared size + sha256) as the transient unit `iq4r-fetch` (log
|
||||
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
|
||||
`.rad` files are mounted.
|
||||
|
||||
**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e`
|
||||
= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's
|
||||
**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest,
|
||||
never the image ID (`69d05724…` is the config digest and sends `podman run` to the network).
|
||||
The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one
|
||||
command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
|
||||
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the
|
||||
`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry.
|
||||
**Runtime** — `stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed`
|
||||
= pin **1.8.1** (adopted 2026-10-18). The version oracle is the boot's own `STARTUP REPORT --
|
||||
radiance 1.8.1` line, because `podman inspect` reports only the digest. The image lives in dank's
|
||||
**rootless** storage and the host needs nothing but the amdgpu driver. Pin the INDEX manifest
|
||||
digest, never the image ID (`69d05724…` is the config digest and sends `podman run` to the
|
||||
network). The serve script's gate is that digest ref itself — `podman image exists name@sha256:…`
|
||||
in one command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
|
||||
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws). **Two CLI breaks ride this pin,
|
||||
and both are boot refusals, never silently-ignored flags:** the `--vram-weights-mib`/`--vram-kv-mib`
|
||||
pair died at 1.3.0 (the resolver derives the split) and `--prefix-cache-dir` /
|
||||
`--prefix-cache-disk-mib` died with the disk tier. A rollback to 1.0.13 (`sha256:4c3f4845…`) is
|
||||
therefore a trio rewrite, not a digest swap — the Eras row carries it, and
|
||||
`tests/frank_archon_unit.bats` asserts the retired knobs are ABSENT from env and argv so a stale
|
||||
line cannot reach a boot.
|
||||
|
||||
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
|
||||
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
|
||||
@@ -64,8 +71,10 @@ the name registered, and every one of those identities holds this GPU. Container
|
||||
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
|
||||
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
|
||||
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
|
||||
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`
|
||||
template as a single-file mount, then `exec podman wait`: the script's lifetime IS the
|
||||
memlock=-1:-1`, `/models` ro + the rad-tune cache at
|
||||
`/opt/radiance/share/radiance/tune` + the template as a single-file mount (there is **no `/data`**
|
||||
any more: it left with the prefix disk tier, and `FRANK_ARCHON_STATE_DIR` is now only the host-side
|
||||
directory that holds `tune/`), then `exec podman wait`: the script's lifetime IS the
|
||||
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
|
||||
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
|
||||
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
|
||||
@@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
|
||||
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|
||||
|---|---|
|
||||
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
|
||||
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
|
||||
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
|
||||
| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
|
||||
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
|
||||
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
|
||||
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
|
||||
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
|
||||
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
|
||||
| Spec / batch | — |
|
||||
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
|
||||
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
|
||||
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
|
||||
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
||||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
|
||||
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
||||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||||
|
||||
Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens**
|
||||
(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab
|
||||
**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision
|
||||
live (≤ 4,194,304 px, 16,384 patches/pass).
|
||||
Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
|
||||
the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
|
||||
`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
|
||||
**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
|
||||
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
|
||||
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
|
||||
(≤ 4,194,304 px, 16,384 patches/pass).
|
||||
|
||||
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
||||
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
||||
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
|
||||
per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32
|
||||
GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the
|
||||
per-boot VRAM planner to second-guess. For comparison, the publisher's own suggested run line on 32
|
||||
GB cards was `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3 — a
|
||||
1.0.x-era line, and `--expert-vs-cache-ratio` was removed at 1.2.0, so today it is a capacity floor
|
||||
quoted from the card, not a run line this pin accepts; the
|
||||
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
|
||||
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
|
||||
one string).
|
||||
@@ -130,6 +145,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
|
||||
|
||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||
|---|---|---|---|---|---|
|
||||
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
|
||||
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
|
||||
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
|
||||
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
|
||||
@@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
|
||||
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
|
||||
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
|
||||
- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
|
||||
- **Pool sizing law (the explicit-pair era — at 1.8.1 the cache is derived, so the resolver law below
|
||||
owns it).** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
|
||||
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
|
||||
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
|
||||
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
|
||||
@@ -174,13 +191,15 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
|
||||
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
|
||||
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
|
||||
its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is
|
||||
Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change.
|
||||
its regression test. A fourth full-height conversation needs 6,601 MiB of pool: refusing it is
|
||||
Frank's explicit call, and under that pin serving it was a `--vram-kv-mib` change, not a code change.
|
||||
At 1.8.1 the same decision is `MAX_SEQS` — the resolver buys the cache the demand signal implies.
|
||||
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
|
||||
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
|
||||
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
|
||||
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
|
||||
- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
|
||||
- **One VRAM budget per card** (the arithmetic below is the pair era's; the ceiling survives, the knobs
|
||||
do not): `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
|
||||
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
|
||||
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
|
||||
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
|
||||
@@ -188,12 +207,34 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
|
||||
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
|
||||
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
|
||||
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
|
||||
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
|
||||
buffers + arena onto the weights side (~29.6 GiB/card effective).
|
||||
engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
|
||||
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
|
||||
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
|
||||
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
|
||||
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
|
||||
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
|
||||
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
|
||||
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
|
||||
hand. Three boots of evidence:
|
||||
|
||||
| shape | demand | cache given | backing |
|
||||
|---|---|---|---|
|
||||
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
|
||||
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
|
||||
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
|
||||
|
||||
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
|
||||
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
|
||||
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
|
||||
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
|
||||
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
|
||||
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
|
||||
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
|
||||
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
|
||||
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
|
||||
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
|
||||
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
|
||||
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today)
|
||||
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
|
||||
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
|
||||
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
|
||||
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
|
||||
@@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
||||
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
||||
GiB, refused).
|
||||
- **Decode pays for the chunk it shares a step with.** Census of the live engine
|
||||
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||||
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
|
||||
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||||
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
||||
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
||||
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
||||
@@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
||||
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
||||
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
||||
[[LLM Performance Diagnosis]] §9.
|
||||
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
|
||||
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
|
||||
steps after each chunk) are absent from 1.0.13: proven on the running container
|
||||
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
|
||||
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
|
||||
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
|
||||
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
|
||||
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
|
||||
re-scores draft acceptance before it is trusted.
|
||||
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
|
||||
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
|
||||
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
|
||||
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
|
||||
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
|
||||
specialization, and radiance booted seqs 2 on its first fp8-native start.
|
||||
**Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the
|
||||
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM —
|
||||
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the
|
||||
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding
|
||||
"doesn't fit"; the container-side procedure and the kernel gate are .
|
||||
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
|
||||
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
|
||||
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
|
||||
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
|
||||
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
|
||||
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
|
||||
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
|
||||
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
|
||||
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
|
||||
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
|
||||
it is trusted.
|
||||
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
|
||||
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
|
||||
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
|
||||
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
|
||||
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
|
||||
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
|
||||
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
|
||||
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
|
||||
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
|
||||
fp8-native start.
|
||||
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
|
||||
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
|
||||
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
|
||||
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
|
||||
"does not fit" without that line has not read the runtime.
|
||||
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
|
||||
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
|
||||
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
|
||||
|
||||
Reference in New Issue
Block a user