docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape
The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the --vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and bats carried the new shape while the page and the AGENTS.md block still described the 1.0.13 3-slot era, so every point-of-use number was wrong. Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not from the stale pre-#31 draft: runtime block (digest, version oracle, the two CLI breaks that are boot refusals), shape table (demand-signal rows, derived VRAM split, restored spec/batch row, host-only prefix tier, step sharing), container contract (no /data mount), the measured 2-slot ledger, a new Eras row for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and the laws: pool sizing and the per-card budget re-scoped to the pair era, a new resolver law (funds ~90% of the worst case, host RAM below a slab step is inert, MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the fix ordered-not-verified, the knob-availability law flipped, and the slots law now pricing the third slot in KV (4.20 -> 2.76 GiB).
This commit is contained in:
@@ -462,12 +462,15 @@ disagrees with it is the bug.
|
|||||||
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
|
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
|
||||||
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
|
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
|
||||||
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
|
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
|
||||||
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
|
`stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
|
||||||
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
|
- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
|
||||||
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
|
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
|
||||||
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
|
MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
|
||||||
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
|
tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
|
||||||
+ 64 GiB disk, n-gram table pinned to RAM.
|
by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
|
||||||
|
the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk
|
||||||
|
tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal,
|
||||||
|
not a no-op.
|
||||||
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
|
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
|
||||||
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
|
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
|
||||||
is why this is a root unit, and `ttm-preflight.sh` raises
|
is why this is a root unit, and `ttm-preflight.sh` raises
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
type: reference
|
type: reference
|
||||||
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)"
|
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.8.1, publisher Qwen3.8-Flash-Next fp8-iq4r-moe .rad, 2× R9700 TP=2 (ACTIVE)"
|
||||||
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
|
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
|
||||||
tags:
|
tags:
|
||||||
- llm
|
- llm
|
||||||
@@ -12,7 +12,7 @@ tags:
|
|||||||
- tp2
|
- tp2
|
||||||
- expert-offload
|
- expert-offload
|
||||||
- endpoint
|
- endpoint
|
||||||
timestamp: 2026-10-18
|
timestamp: 2026-10-19
|
||||||
---
|
---
|
||||||
|
|
||||||
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
|
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
|
||||||
@@ -20,8 +20,9 @@ timestamp: 2026-10-18
|
|||||||
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
|
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
|
||||||
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
|
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
|
||||||
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
|
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
|
||||||
PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap
|
PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
|
||||||
is a `MODEL_FILE` line plus a re-download — see Eras.
|
is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws).
|
||||||
|
A payload swap is a `MODEL_FILE` line plus a re-download — see Eras.
|
||||||
|
|
||||||
## Current shape
|
## Current shape
|
||||||
|
|
||||||
@@ -44,14 +45,20 @@ declared size + sha256) as the transient unit `iq4r-fetch` (log
|
|||||||
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
|
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
|
||||||
`.rad` files are mounted.
|
`.rad` files are mounted.
|
||||||
|
|
||||||
**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e`
|
**Runtime** — `stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed`
|
||||||
= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's
|
= pin **1.8.1** (adopted 2026-10-18). The version oracle is the boot's own `STARTUP REPORT --
|
||||||
**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest,
|
radiance 1.8.1` line, because `podman inspect` reports only the digest. The image lives in dank's
|
||||||
never the image ID (`69d05724…` is the config digest and sends `podman run` to the network).
|
**rootless** storage and the host needs nothing but the amdgpu driver. Pin the INDEX manifest
|
||||||
The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one
|
digest, never the image ID (`69d05724…` is the config digest and sends `podman run` to the
|
||||||
command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
|
network). The serve script's gate is that digest ref itself — `podman image exists name@sha256:…`
|
||||||
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the
|
in one command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
|
||||||
`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry.
|
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws). **Two CLI breaks ride this pin,
|
||||||
|
and both are boot refusals, never silently-ignored flags:** the `--vram-weights-mib`/`--vram-kv-mib`
|
||||||
|
pair died at 1.3.0 (the resolver derives the split) and `--prefix-cache-dir` /
|
||||||
|
`--prefix-cache-disk-mib` died with the disk tier. A rollback to 1.0.13 (`sha256:4c3f4845…`) is
|
||||||
|
therefore a trio rewrite, not a digest swap — the Eras row carries it, and
|
||||||
|
`tests/frank_archon_unit.bats` asserts the retired knobs are ABSENT from env and argv so a stale
|
||||||
|
line cannot reach a boot.
|
||||||
|
|
||||||
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
|
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
|
||||||
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
|
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
|
||||||
@@ -64,8 +71,10 @@ the name registered, and every one of those identities holds this GPU. Container
|
|||||||
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
|
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
|
||||||
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
|
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
|
||||||
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
|
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
|
||||||
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`
|
memlock=-1:-1`, `/models` ro + the rad-tune cache at
|
||||||
template as a single-file mount, then `exec podman wait`: the script's lifetime IS the
|
`/opt/radiance/share/radiance/tune` + the template as a single-file mount (there is **no `/data`**
|
||||||
|
any more: it left with the prefix disk tier, and `FRANK_ARCHON_STATE_DIR` is now only the host-side
|
||||||
|
directory that holds `tune/`), then `exec podman wait`: the script's lifetime IS the
|
||||||
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
|
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
|
||||||
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
|
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
|
||||||
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
|
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
|
||||||
@@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
|
|||||||
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|
||||||
|---|---|
|
|---|---|
|
||||||
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
|
||||||
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
|
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
|
||||||
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
|
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
|
||||||
| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
|
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
|
||||||
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
|
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
|
||||||
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
|
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
|
||||||
| Spec / batch | — |
|
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
|
||||||
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
|
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
|
||||||
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
||||||
|
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
|
||||||
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
||||||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||||||
|
|
||||||
Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens**
|
Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
|
||||||
(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab
|
the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
|
||||||
**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision
|
`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
|
||||||
live (≤ 4,194,304 px, 16,384 patches/pass).
|
**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
|
||||||
|
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
|
||||||
|
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
|
||||||
|
(≤ 4,194,304 px, 16,384 patches/pass).
|
||||||
|
|
||||||
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
||||||
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
||||||
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
|
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
|
||||||
per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32
|
per-boot VRAM planner to second-guess. For comparison, the publisher's own suggested run line on 32
|
||||||
GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the
|
GB cards was `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3 — a
|
||||||
|
1.0.x-era line, and `--expert-vs-cache-ratio` was removed at 1.2.0, so today it is a capacity floor
|
||||||
|
quoted from the card, not a run line this pin accepts; the
|
||||||
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
|
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
|
||||||
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
|
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
|
||||||
one string).
|
one string).
|
||||||
@@ -130,6 +145,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
|
|||||||
|
|
||||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
|
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
|
||||||
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
|
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
|
||||||
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
|
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
|
||||||
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
|
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
|
||||||
@@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
|
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
|
||||||
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
|
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
|
||||||
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
|
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
|
||||||
- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
|
- **Pool sizing law (the explicit-pair era — at 1.8.1 the cache is derived, so the resolver law below
|
||||||
|
owns it).** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
|
||||||
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
|
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
|
||||||
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
|
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
|
||||||
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
|
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
|
||||||
@@ -174,13 +191,15 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
|
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
|
||||||
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
|
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
|
||||||
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
|
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
|
||||||
its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is
|
its regression test. A fourth full-height conversation needs 6,601 MiB of pool: refusing it is
|
||||||
Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change.
|
Frank's explicit call, and under that pin serving it was a `--vram-kv-mib` change, not a code change.
|
||||||
|
At 1.8.1 the same decision is `MAX_SEQS` — the resolver buys the cache the demand signal implies.
|
||||||
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
|
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
|
||||||
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
|
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
|
||||||
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
|
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
|
||||||
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
|
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
|
||||||
- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
|
- **One VRAM budget per card** (the arithmetic below is the pair era's; the ceiling survives, the knobs
|
||||||
|
do not): `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
|
||||||
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
|
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
|
||||||
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
|
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
|
||||||
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
|
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
|
||||||
@@ -188,12 +207,34 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
|
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
|
||||||
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
|
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
|
||||||
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
|
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
|
||||||
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
|
engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
|
||||||
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
|
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
|
||||||
buffers + arena onto the weights side (~29.6 GiB/card effective).
|
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
|
||||||
|
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
|
||||||
|
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
|
||||||
|
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
|
||||||
|
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
|
||||||
|
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
|
||||||
|
hand. Three boots of evidence:
|
||||||
|
|
||||||
|
| shape | demand | cache given | backing |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
|
||||||
|
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
|
||||||
|
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
|
||||||
|
|
||||||
|
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
|
||||||
|
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
|
||||||
|
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
|
||||||
|
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
|
||||||
|
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
|
||||||
|
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
|
||||||
|
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
|
||||||
|
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
|
||||||
|
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
|
||||||
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
|
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
|
||||||
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
|
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
|
||||||
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today)
|
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
|
||||||
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
|
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
|
||||||
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
|
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
|
||||||
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
|
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
|
||||||
@@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
||||||
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
||||||
GiB, refused).
|
GiB, refused).
|
||||||
- **Decode pays for the chunk it shares a step with.** Census of the live engine
|
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
|
||||||
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||||||
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
||||||
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
||||||
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
||||||
@@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
|||||||
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
||||||
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
||||||
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
||||||
[[LLM Performance Diagnosis]] §9.
|
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
|
||||||
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
|
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
|
||||||
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
|
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
|
||||||
steps after each chunk) are absent from 1.0.13: proven on the running container
|
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
|
||||||
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
|
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
|
||||||
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
|
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
|
||||||
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
|
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
|
||||||
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
|
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
|
||||||
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
|
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
|
||||||
re-scores draft acceptance before it is trusted.
|
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
|
||||||
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
|
it is trusted.
|
||||||
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
|
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||||||
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
|
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
|
||||||
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
|
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
|
||||||
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
|
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
|
||||||
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
|
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
|
||||||
specialization, and radiance booted seqs 2 on its first fp8-native start.
|
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
|
||||||
**Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the
|
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
|
||||||
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM —
|
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
|
||||||
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the
|
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
|
||||||
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding
|
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
|
||||||
"doesn't fit"; the container-side procedure and the kernel gate are .
|
fp8-native start.
|
||||||
|
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
|
||||||
|
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
|
||||||
|
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
|
||||||
|
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
|
||||||
|
"does not fit" without that line has not read the runtime.
|
||||||
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
|
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
|
||||||
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
|
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
|
||||||
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
|
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
|
||||||
|
|||||||
Reference in New Issue
Block a user