@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152`**per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…`**plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and**`.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len`×`--max-num-seqs`) and
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
hand. Three boots of evidence:
- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
`core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
`/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
and the 400 class is a single request over `--max-model-len`.
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -366,17 +389,18 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
`--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
`interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
fp8-native start.
- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
`seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
"seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
specialization, and radiance booted seqs 2 on its first fp8-native start.
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -570,13 +594,13 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
longer than the KV pool's token count is accepted then refused as too large to fit; a request
beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and
answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height
sessions, admitting a second near-full-height conversation can **preempt a running sequence from
the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so
its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current
boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative
count and `radiance:num_requests_preempted` only the instant depth.
beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
sessions, so four conversations can be resident but a fourth near-full-height one can still
**preempt a running sequence from the back** to free its blocks
(`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
`radiance:num_requests_preempted` is only the instant depth.
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.