Compare commits

..
Author SHA1 Message Date
Frank 1aeb7af6df docs(probe): price the corpus chars/token ratio at run time
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
2026-10-20 23:26:53 -07:00
Frank cd0be97b39 docs(vault): the four-slot ledger, and the resolver's funding law from the source
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.

The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
2026-10-20 23:26:53 -07:00
Frank 1bb6399524 feat(frank-archon): four decode slots over a >=700k-token KV pool
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.

Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.

TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
2026-10-20 23:10:05 -07:00
5 changed files with 125 additions and 82 deletions

No files matched your search

+27 -22
View File
@@ -46,31 +46,35 @@ FRANK_ARCHON_TP=2
# stays EXACT for single-sequence decode and any step of <= 25 rows — see # stays EXACT for single-sequence decode and any step of <= 25 rows — see
# MAX_SEQS x (1 + SPEC_TOKENS) below. # MAX_SEQS x (1 + SPEC_TOKENS) below.
FRANK_ARCHON_TP_WIRE=wht6 FRANK_ARCHON_TP_WIRE=wht6
# Decode slots. A slot costs KV, never decode; the pool is sized to this # Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
# contract and nothing more. At 1.8.1 the budget resolver caps the cache at # exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height # retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
# conversation answers 400 while /health and every shorter chat work fine # least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two # target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane, # the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
# instead of advertising a third slot that only exists at shorter prompts. # keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
FRANK_ARCHON_MAX_SEQS=2 # 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
# fourth conversations are backed by the cache instead of over-committing it.
FRANK_ARCHON_MAX_SEQS=4
# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD # Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
# limit, and models.yml contextWindow MUST quote this number (over-length is a # limit, and models.yml contextWindow MUST quote this number (over-length is a
# refusal, not a truncation). The pool is sized for seqs x maxlen, not one # refusal, not a truncation). The pool is sized for seqs x maxlen, not one
# sequence. # sequence.
FRANK_ARCHON_MAX_MODEL_LEN=200000 FRANK_ARCHON_MAX_MODEL_LEN=200000
FRANK_ARCHON_PLACEMENT=expert_tiered FRANK_ARCHON_PLACEMENT=expert_tiered
# Pinned host pool for expert rows that do not fit VRAM. Under the budget # Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
# resolver this is ALSO the lever that decides which experts stay on the card: # host must pin tp x this number, and the refusal that names it is
# the card's leftover goes to the expert plane first and the KV cache takes the # core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
# rest, so growing the pool moves experts off the card and shrinking it spills # lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864 # MiB of experts off the card and into the cache. Derived from the boot ledger (a
# refused a boot). The ledger's coverage line is the referee, not this file. # 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left # drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra # the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
# host RAM above what the expert plane's footprint needs buys no cache. Back at # twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
# 44,928, the size that boots clean. # residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
FRANK_ARCHON_HOST_POOL_MIB=44928 # x8 link when the router picks it. The ledger's coverage line is the referee, not
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
FRANK_ARCHON_HOST_POOL_MIB=49152
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle # VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
# floor is built from it. # floor is built from it.
FRANK_ARCHON_GPU_HEADROOM_MIB=96 FRANK_ARCHON_GPU_HEADROOM_MIB=96
@@ -131,9 +135,10 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies # RESETS pages_limit to half of RAM on every boot; below the plan the load dies
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob # mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
# itself before refusing by name, so recovery is a boot, not a visit. # itself before refusing by name, so recovery is a boot, not a visit.
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT # Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB. # auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
FRANK_ARCHON_TTM_MIN_MIB=150000 # at the 49,152 pool with 4 slots that computes to 161,848 MiB.
FRANK_ARCHON_TTM_MIN_MIB=162000
FRANK_ARCHON_TTM_TARGET_MIB=204800 FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands # Escape hatch for experiments. Unset -> no extra argv; the serve script expands
# this unquoted on purpose, so keep multi-flag values as one string. # this unquoted on purpose, so keep multi-flag values as one string.
+16 -5
View File
@@ -147,10 +147,21 @@ def main():
deep = corpus(160_000) deep = corpus(160_000)
wide = corpus(1_100_000) wide = corpus(1_100_000)
# 3.28 chars/token is the measured ratio of this corpus through the # The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11 # (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
# deploy); 98.5 % of the window leaves the response room under the ceiling. # 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
chars = int(0.985 * maxlen * 3.28) # stale constant sends leg B past --max-model-len, where the engine answers a
# perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
# context size (200000)") and the leg reads as a capacity failure. One small
# request prices the ratio; 97 % of the window leaves the response room.
sample = prompt(wide, 4000, "CAL", "cal-marker")
cal, _, _ = post({"model": MODEL,
"messages": [{"role": "user", "content": sample}],
"max_tokens": 1, "temperature": 0.0,
"chat_template_kwargs": {"enable_thinking": False}})
ratio = len(sample) / cal["usage"]["prompt_tokens"]
print(f"[cal] corpus ratio {ratio:.2f} chars/token")
chars = int(0.97 * maxlen * ratio)
# Each marker's FIRST hyphen-separated word is unique, because show() # Each marker's FIRST hyphen-separated word is unique, because show()
# scores the hit on that word — a shared prefix would score every slot HIT # scores the hit on that word — a shared prefix would score every slot HIT
# on any answer that says the word. # on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}") print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s") print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth. # D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28) dchars = int(0.25 * maxlen * ratio)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512) dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)] for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, " print(f"[D] {seqs} concurrent streams x 512 tokens, "
+6 -6
View File
@@ -15,9 +15,9 @@
# log states every budget before anything is allocated (weights vs KV # log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin, # split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize. # the P2P answer per pair) — read it after every resize.
# Deployed shape (2026-10-19, radiance 1.8.1): TWO decode slots x 200,000 tokens, # Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
# fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in pinned host # >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
# RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV # in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget # pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
# resolver derives it from headroom, the cache bound and the host pool, so the slab # resolver derives it from headroom, the cache bound and the host pool, so the slab
# size and the pool's token count are LEDGER lines. After every boot read them, plus # size and the pool's token count are LEDGER lines. After every boot read them, plus
@@ -27,9 +27,9 @@
# pooled experts are read synchronously) — and the cache's session backing. # pooled experts are read synchronously) — and the cache's session backing.
# #
# The root unit (scripts/frank-archon.service, User=dank, # The root unit (scripts/frank-archon.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools + # LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned # table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
# pages, which is ALSO why this script refuses to start when ttm's # pinned pages, which is ALSO why this script refuses to start when ttm's
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below). # pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
set -euo pipefail set -euo pipefail
+69 -45
View File
@@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value | | Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---| |---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default | | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation | | Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) | | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) | | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** | | Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) | | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT | | Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`): Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`):
the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**, the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**,
`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages `prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`.
**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`.
(≤ 4,194,304 px, 16,384 patches/pass). `--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned
host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows**
(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live
(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory
directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of
the bigger cache, paid over PCIe when those experts are hit.
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin, anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
@@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti
| Window | Payload | Runtime | Shape | Replaced because | Rollback | | Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` |
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) | | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) | | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) | | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
@@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:** buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal. the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.** - **At 1.8.1 the split belongs to the resolver, and the source fixes its order.**
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena →
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by floor, not a share**, and divides the `elastic` remainder in three lines (146-156):
hand. Three boots of evidence:
| shape | demand | cache given | backing | ```cpp
exp = std::min(b.expert_floor, b.elastic);
kv = std::min(kv_cap, std::max(b.elastic - exp, kv_one));
exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv);
```
`kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes`
(`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never
its funding. The funding is `elastic − expert_floor`, and the floor is
`min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with
`host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the
cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache
takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card),
which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of
evidence, all on 1.8.1:
| shape | demand | cache given | engine's `backs %` |
|---|---|---|---| |---|---|---|---|
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) | | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % |
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % | | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) | | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % |
| 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % |
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler — The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle. window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe —
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8 (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` — per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape:
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache. `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused,
and the 400 class is a single request over `--max-model-len`.
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275 - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era) 256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
@@ -366,17 +389,18 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
`--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest
`interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at
interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes. interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes.
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it - **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6 `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic −
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger 2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the 2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old
fp8-native start. "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
specialization, and radiance booted seqs 2 on its first fp8-native start.
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
pinned weights, and the doubling is invisible in the weight count. Read the engine log for pinned weights, and the doubling is invisible in the weight count. Read the engine log for
@@ -570,13 +594,13 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`. — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
longer than the KV pool's token count is accepted then refused as too large to fit; a request longer than the KV pool's token count is accepted then refused as too large to fit; a request
beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here)
answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height
sessions, admitting a second near-full-height conversation can **preempt a running sequence from sessions, so four conversations can be resident but a fourth near-full-height one can still
the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so **preempt a running sequence from the back** to free its blocks
its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then
boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`);
count and `radiance:num_requests_preempted` only the instant depth. `radiance:num_requests_preempted` is only the instant depth.
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the
netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under
+7 -4
View File
@@ -145,7 +145,7 @@ nosudo_bin() {
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted # (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3 # seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
# tied inside the +-3% noise floor — slots buy concurrency, not speed. # tied inside the +-3% noise floor — slots buy concurrency, not speed.
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_MAX_SEQS=4$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE" grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE" grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
@@ -158,7 +158,7 @@ nosudo_bin() {
# so the pool's token count is a boot-ledger line — verified live by the # so the pool's token count is a boot-ledger line — verified live by the
# capacity leg of scripts/probe-frank-archon-kv.py at every cutover. # capacity leg of scripts/probe-frank-archon-kv.py at every cutover.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE" grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE" grep -q '^FRANK_ARCHON_HOST_POOL_MIB=49152$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE" grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE" grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache: pinned host RAM only (the disk tier was removed), and the # Prefix cache: pinned host RAM only (the disk tier was removed), and the
@@ -177,8 +177,11 @@ nosudo_bin() {
[ "$ttmt" -gt "$ttmm" ] [ "$ttmt" -gt "$ttmm" ]
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE") pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE") ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_CACHE_MIB=/{print $2}' "$ENV_FILE")
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up. # 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host + MNBT + the auto
ledger=$(( 2 * pool + 48824 + ph + 2048 )) # checkpoint store (MAX_SEQS x 10 slots of 112 MiB -- the store is pinned host
# memory too, and it grows with the slot count).
seqs=$(awk -F= '/^FRANK_ARCHON_MAX_SEQS=/{print $2}' "$ENV_FILE")
ledger=$(( 2 * pool + 48824 + ph + 2048 + seqs * 10 * 112 ))
[ "$ttmm" -ge "$ledger" ] [ "$ttmm" -ge "$ledger" ]
# The .rad size floor is positive (a truncated convert must fail fast). # The .rad size floor is positive (a truncated convert must fail fast).
grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE" grep -q '^FRANK_ARCHON_RAD_MIN_MIB=' "$ENV_FILE"