`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
@@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---|
|---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM pair |`VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming`VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | — |
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…`**plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and**`.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and**`.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
@@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective).
buffers + arena onto the weights side (~29.6 GiB/card effective).**Neither flag exists at 1.8.1:**
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len`×`--max-num-seqs`) and
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
hand. Three boots of evidence:
| shape | demand | cache given | backing |
|---|---|---|---|
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today)
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
@@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
@@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9.
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
steps after each chunk) are absent from 1.0.13: proven on the running container
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
re-scores draft acceptance before it is trusted.
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
it is trusted.
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
specialization, and radiance booted seqs 2 on its first fp8-native start.
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
**Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM —
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
"doesn't fit"; the container-side procedure and the kernel gate are .
fp8-native start.
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
"does not fit" without that line has not read the runtime.
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.