From d9b82b372e31930c6dd9dc7cc069528590e08ef1 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 14:55:43 -0700 Subject: [PATCH 1/2] docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the --vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and bats carried the new shape while the page and the AGENTS.md block still described the 1.0.13 3-slot era, so every point-of-use number was wrong. Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not from the stale pre-#31 draft: runtime block (digest, version oracle, the two CLI breaks that are boot refusals), shape table (demand-signal rows, derived VRAM split, restored spec/batch row, host-only prefix tier, step sharing), container contract (no /data mount), the measured 2-slot ledger, a new Eras row for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and the laws: pool sizing and the per-card budget re-scoped to the pair era, a new resolver law (funds ~90% of the worst case, host RAM below a slab step is inert, MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the fix ordered-not-verified, the knob-availability law flipped, and the slots law now pricing the third slot in KV (4.20 -> 2.76 GiB). --- AGENTS.md | 15 +- serverAppConfigs-docs/LLM/frank-archon.md | 160 ++++++++++++++-------- 2 files changed, 112 insertions(+), 63 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a146aa1..4561d0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -462,12 +462,15 @@ disagrees with it is the bug. - Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template `scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned - `stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name. -- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`, - `expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair - `--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤ - 30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host - + 64 GiB disk, n-gram table pinned to RAM. + `stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name. +- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`, + `expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV, + MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host + tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived + by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and + the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk + tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal, + not a no-op. - Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000 (over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger is why this is a root unit, and `ttm-preflight.sh` raises diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 1dc6542..6c5a3ab 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -1,6 +1,6 @@ --- type: reference -title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)" +title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.8.1, publisher Qwen3.8-Flash-Next fp8-iq4r-moe .rad, 2× R9700 TP=2 (ACTIVE)" description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught." tags: - llm @@ -12,7 +12,7 @@ tags: - tp2 - expert-offload - endpoint -timestamp: 2026-10-18 +timestamp: 2026-10-19 --- # frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770 @@ -20,8 +20,9 @@ timestamp: 2026-10-18 The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving path) driving `podman run` straight from the digest-pinned image under a **root** systemd unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI -PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap -is a `MODEL_FILE` line plus a re-download — see Eras. +PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair +is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws). +A payload swap is a `MODEL_FILE` line plus a re-download — see Eras. ## Current shape @@ -44,14 +45,20 @@ declared size + sha256) as the transient unit `iq4r-fetch` (log `/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only `.rad` files are mounted. -**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e` -= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's -**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest, -never the image ID (`69d05724…` is the config digest and sends `podman run` to the network). -The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one -command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer -`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the -`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry. +**Runtime** — `stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed` += pin **1.8.1** (adopted 2026-10-18). The version oracle is the boot's own `STARTUP REPORT -- +radiance 1.8.1` line, because `podman inspect` reports only the digest. The image lives in dank's +**rootless** storage and the host needs nothing but the amdgpu driver. Pin the INDEX manifest +digest, never the image ID (`69d05724…` is the config digest and sends `podman run` to the +network). The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` +in one command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer +`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws). **Two CLI breaks ride this pin, +and both are boot refusals, never silently-ignored flags:** the `--vram-weights-mib`/`--vram-kv-mib` +pair died at 1.3.0 (the resolver derives the split) and `--prefix-cache-dir` / +`--prefix-cache-disk-mib` died with the disk tier. A rollback to 1.0.13 (`sha256:4c3f4845…`) is +therefore a trio rewrite, not a digest swap — the Eras row carries it, and +`tests/frank_archon_unit.bats` asserts the retired knobs are ABSENT from env and argv so a stale +line cannot reach a boot. **Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header) `scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image @@ -64,8 +71,10 @@ the name registered, and every one of those identities holds this GPU. Container (publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring — the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit -memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune` -template as a single-file mount, then `exec podman wait`: the script's lifetime IS the +memlock=-1:-1`, `/models` ro + the rad-tune cache at +`/opt/radiance/share/radiance/tune` + the template as a single-file mount (there is **no `/data`** +any more: it left with the prefix disk tier, and `FRANK_ARCHON_STATE_DIR` is now only the host-side +directory that holds `tune/`), then `exec podman wait`: the script's lifetime IS the container's, so a crash is a nonzero `ExecStart` and the restart policy is real. Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank` `LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the @@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Knob (env name, `FRANK_ARCHON_` dropped) | Live value | |---|---| | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default | -| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) | -| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) | -| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` | +| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation | +| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) | +| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) | | Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** | | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) | -| Spec / batch | — | -| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) | +| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | +| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | +| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | -Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens** -(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab -**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision -live (≤ 4,194,304 px, 16,384 patches/pass). +Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`): +the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**, +`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages +**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the +2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots +(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live +(≤ 4,194,304 px, 16,384 patches/pass). The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin, the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no -per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32 -GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the +per-boot VRAM planner to second-guess. For comparison, the publisher's own suggested run line on 32 +GB cards was `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3 — a +1.0.x-era line, and `--expert-vs-cache-ratio` was removed at 1.2.0, so today it is a capacity floor +quoted from the card, not a run line this pin accepts; the shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as one string). @@ -130,6 +145,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| +| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) | | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) | | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) | | 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account | @@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). 2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer` lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16. -- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype- +- **Pool sizing law (the explicit-pair era — at 1.8.1 the cache is derived, so the resolver law below + owns it).** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype- independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/ tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 − @@ -174,13 +191,15 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is - its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is - Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change. + its regression test. A fourth full-height conversation needs 6,601 MiB of pool: refusing it is + Frank's explicit call, and under that pin serving it was a `--vram-kv-mib` change, not a code change. + At 1.8.1 the same decision is `MAX_SEQS` — the resolver buys the cache the demand signal implies. - **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a HARD limit — "one request longer than N tokens is accepted by the API and then refused as too large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for `seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size. -- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget +- **One VRAM budget per card** (the arithmetic below is the pair era's; the ceiling survives, the knobs + do not): `weights + kv ≤ 30,969 MiB`, read off the engine's own budget table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and "*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and `--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with @@ -188,12 +207,34 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849 = 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120 ("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the - engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never - `--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight - buffers + arena onto the weights side (~29.6 GiB/card effective). + engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, + never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight + buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:** + the same ceiling is enforced by the resolver, and passing the pair is a boot refusal. +- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.** + The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and + `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and + sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by + hand. Three boots of evidence: + + | shape | demand | cache given | backing | + |---|---|---|---| + | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) | + | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % | + | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) | + + So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler — + never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a + slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is + grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle. + `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot + hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8 + era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` — + the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over + `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache. - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275 slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= - 256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today) + 256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era) so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 % decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost @@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). - **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3 GiB, refused). -- **Decode pays for the chunk it shares a step with.** Census of the live engine - (radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot): +- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the + 3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot): 1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every @@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method: - [[LLM Performance Diagnosis]] §9. -- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and - `--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only - steps after each chunk) are absent from 1.0.13: proven on the running container - (`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the - 2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is - an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the - stall, and this page's own MNBT law already knows 2560 collapses decode (MTP - resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try - re-scores draft acceptance before it is trusted. -- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it - adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs × - (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step - of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06: - seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2 - cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape - specialization, and radiance booted seqs 2 on its first fp8-native start. - **Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the - EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM — - the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the - weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding - "doesn't fit"; the container-side procedure and the kernel gate are . + [[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law), + and nobody has re-run the census since the migration — so the fix is ordered, **not verified**. +- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the + shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks; + the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk + step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until + the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration. + The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a + knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an + edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before + it is trusted. +- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it + adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), + declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows + (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6 + today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % + floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it + moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while + the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger + mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the + retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first + fp8-native start. +- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request + of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the + pinned weights, and the doubling is invisible in the weight count. Read the engine log for + `Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload + "does not fit" without that line has not read the runtime. - **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable. - **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)` -- 2.54.0 From 7eed636bd13e4ba153ff5e2e9eb676a5028bb603 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 14:55:43 -0700 Subject: [PATCH 2/2] =?UTF-8?q?docs(perf):=20destroy=20=C2=A79's=20vendor?= =?UTF-8?q?=20absolutes,=20keep=20the=20ratios?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The t/s ladder and the TTFT delta triple measure a configuration neither deployed nor compared; per the purpose test they are dead weight. What survives is the knob pricing (halving the shared chunk costs roughly a fifth of prefill throughput), the long-prompt-only shape of the span win, and the method. Run-on now names the era the census was taken in. --- .../LLM/LLM Performance Diagnosis.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index ca0647e..3ba2114 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -251,21 +251,20 @@ Knobs, in the order worth trying them (radiance GUIDE §4.5): the prefill stager's smallest pass — the startup line names it, and below it every routed layer reads its pooled experts synchronously, which costs the prefill far more than sharing saves. Counts as `engine_prefill_chunks_capped_total`. -- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The - vendor's curve on a 48-layer MoE at 200k on two cards: chunk 512 → 1750 t/s, 1024 - → 2200, 2048 → 2706 (a 185k prompt: 138 s against 91 s), so halving the stall - costs ~19 % of prefill. Raising it past the interval buys nothing and loses expert - residency to the arena. -- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step - while every shared step is planned at a smaller shared budget, which on an - expert-tiered box lends that arena back to the expert slab. Vendor TP2 - expert-tiered TTFT deltas vs a plain 2048 budget: −6.1 % at 2k, +2.3 % at 18k, - +49.1 % at 250k — long prompts only. +- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The vendor's own + curve prices **halving the shared chunk at roughly a fifth of prefill throughput** (a 48-layer MoE + at 200k on two cards — knob pricing, not this endpoint's number, so treat it as the order of + magnitude a chunk decision costs). Raising it past the checkpoint interval buys nothing and loses + expert residency to the arena. +- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step while every + shared step is planned at a smaller shared budget, which on an expert-tiered box lends that arena + back to the expert slab. The vendor's own measurement says the win is **long-prompt-only**: nothing + at short prompts, a large TTFT gain at the far end of the window. Two traps. **The arena/slab coupling makes a shared step doubly expensive on an -expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would -have had (320 MiB/card at TP2 for a 4096 budget), so the decode rows inside that step -also read experts over the link — a smaller shared budget helps twice. And **re-check +expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would have had, so the +decode rows inside that step also read experts over the link — a smaller shared budget helps twice. +And **re-check `prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV pool are the same memory, and the failure looks like turn two taking as long as turn one. @@ -276,8 +275,9 @@ Availability is a version question, not a config question: `--busy-prefill-chunk tree, and check the source clone's depth before dating a flag from `git log -S` (a `--depth 1` clone reports its own tip for every search). -Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = interval -2048) — the measured census and the verdict for this fleet's shape are that page's. +Run on: [[frank-archon]] — the census was taken on its 3-slot radiance 1.0.13 era (budget 2048 = +interval 2048); the live 1.8.1 shape carries the two chunk knobs. The measured census and the verdict +for this fleet's shape are that page's. ## Related -- 2.54.0