`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
@@ -79,27 +88,33 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
| VRAM pair |`VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming`VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | — |
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…`**plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and**`.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
@@ -161,7 +177,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective).
engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective).**Neither flag exists at 1.8.1:**
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len`×`--max-num-seqs`) and
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
hand. Three boots of evidence:
| shape | demand | cache given | backing |
|---|---|---|---|
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today)
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
@@ -239,8 +280,8 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
- **Decode pays for the chunk it shares a step with.** Census of the live engine
(radiance 1.0.13, `/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
@@ -252,28 +293,33 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
[[LLM Performance Diagnosis]] §9.
- **The knobs for it are not in this pin.** `--busy-prefill-chunk` and
`--prefill-interleave` (radiance GUIDE §4.5 — a shared chunk's cap, and N decode-only
steps after each chunk) are absent from 1.0.13: proven on the running container
(`podman exec frank-archon radiance --help` prints no `prefill`/`chunk` flag), and the
2026-10-18 source tree carries both. The only crank this pin has is MNBT, and that is
an arm and not an edit: the vendor prices a 1024 chunk at −19 % prefill to halve the
stall, and this page's own MNBT law already knows 2560 collapses decode (MTP
resonance). An image bump is gated by the 1.2.0 numerics dead end below, so any try
re-scores draft acceptance before it is trusted.
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
specialization, and radiance booted seqs 2 on its first fp8-native start.
**Same era, separate law: a dequantize-to-compute path bills host RAM twice.** On the
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM —
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding
"doesn't fit"; the container-side procedure and the kernel gate are .
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
it is trusted.
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
fp8-native start.
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
"does not fit" without that line has not read the runtime.
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.