diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 2266c2c..513786d 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -48,7 +48,7 @@ FRANK_ARCHON_TP=2 FRANK_ARCHON_TP_WIRE=wht6 # Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is # exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the -# retention policy keeps) per slot. Ordered 2026-10-21: FOUR slots over a pool of at +# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at # least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The # target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether # the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must diff --git a/scripts/serve-frank-archon.sh b/scripts/serve-frank-archon.sh index 708214a..08416ee 100755 --- a/scripts/serve-frank-archon.sh +++ b/scripts/serve-frank-archon.sh @@ -15,7 +15,7 @@ # log states every budget before anything is allocated (weights vs KV # split, the expert host pool's coverage of the worst case, the PLE pin, # the P2P answer per pair) — read it after every resize. -# Deployed shape (2026-10-21, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a +# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a # >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache # in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV # pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 9027e47..cfe0443 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -88,25 +88,30 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Knob (env name, `FRANK_ARCHON_` dropped) | Live value | |---|---| | TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default | -| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation | +| Slots / window | `MAX_SEQS=4`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**: the resolver sizes the KV carve against `slots × window`, and no slot count funds itself (Laws) | | KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) | | VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) | -| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** | +| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=49152` **per rank** — the pinned host pool is also the **KV lever**: the expert floor is `plane + 2 units − host room`, so every MiB pinned there lowers what the experts claim on the card one-for-one and the cache takes it (Laws) | | PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) | | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | | Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT | -| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | +| TTM gate | `TTM_MIN_MIB=162000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | -Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`): -the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**, -`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages -**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the -2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots -(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live -(≤ 4,194,304 px, 16,384 patches/pass). +Boot ledger for the live shape (2026-10-20, pin 1.8.1, `MAX_SEQS=4`, `HOST_POOL_MIB=49152`): +the resolver prints `elastic 26.73 GiB` → **experts 20.72 GiB (78 %) / kv cache 5.64 GiB (21 %)**, +`prefill staging 373.65 MiB` out of the cache, `vram_weights 24.30 GiB`, `host_pool 48.59 GiB`. +The referee line: `kv: the cache can grow to 5.64 GiB a card — 747,492 tokens of context, 3.7 +sessions at 200,000 — with every block it lends the experts taken back; 440.04 MiB of that is +per-sequence state, and up to 5.20 GiB is lent to the experts while no request holds it`. +`--checkpoint-slots 40` is auto (`max_num_seqs × 10` slots of 111.98 MiB = 4,479 MiB of pinned +host), `/health` reports `queue_cap 32` (`max(8, seqs × 8)`), sampler **12 rows** +(= `seqs × (1 + speculative depth)`), 4 hardware queues a card, template 28,234 B, vision live +(≤ 4,194,304 px, 16,384 patches/pass), P2P confirmed (`the 2 cards reach each other's memory +directly`). Experts living in the pinned host pool: **8.99 GiB a rank** (was 5.99) — the price of +the bigger cache, paid over PCIe when those experts are hit. The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin, @@ -145,6 +150,7 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| +| 2026-10-19 → 10-20 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.8.1**, `sha256:d590e300…` (same digest) | 2 × 200,000 fp8 KV — resolver cache 2.76 GiB = 365,204 tokens (91 % backing); host pool 44,928 MiB a rank, `TTM_MIN_MIB=150000`, `queue_cap 16`, sampler 9 rows; `--prefill-interleave 1` armed on 10-20 | Frank ordered **four** concurrent conversations over a ≥ 700,000-token pool; the resolver funds that only by lowering the expert floor with more pinned host room (Laws) | revert `1bb6399` and redeploy `scripts/deploy.sh dankserver --service frank-archon` — trio back to `MAX_SEQS=2` / `HOST_POOL_MIB=44928` / `TTM_MIN_MIB=150000` | | 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) | | 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) | | 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) | @@ -211,27 +217,44 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:** the same ceiling is enforced by the resolver, and passing the pair is a boot refusal. -- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.** - The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and - `--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and - sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by - hand. Three boots of evidence: +- **At 1.8.1 the split belongs to the resolver, and the source fixes its order.** + `core/mem/vram_budget.cpp` claims the card bottom-up (`capacity → already held → drift → arena → + staging → queues → prefetch → headroom = claimable`), takes the **static weights off the top as a + floor, not a share**, and divides the `elastic` remainder in three lines (146-156): - | shape | demand | cache given | backing | + ```cpp + exp = std::min(b.expert_floor, b.elastic); + kv = std::min(kv_cap, std::max(b.elastic - exp, kv_one)); + exp = std::min(std::max(b.expert_ceiling, b.expert_floor), b.elastic - kv); + ``` + + `kv_cap` is `KVManager::ceiling()` = `state + Σ_paged max_seqs × ceil(max_ctx / block) × bytes` + (`core/mem/kv.cpp:266`), so `--max-num-seqs` and `--max-model-len` set the cache's **demand**, never + its funding. The funding is `elastic − expert_floor`, and the floor is + `min(plane + spare, plane + spare + 2·unit_max − host_room)` (`core/place/planner.cpp:100-104`, with + `host_room = host_pool − file_stage_reserve` at `core/engine_bringup.cpp:1418-1422`). **That is the + cache's lever: every MiB added to `--host-pool-mib` drops the expert floor one-for-one and the cache + takes it** — minus `stage`, the prefill stager's ring (two layers of the experts left off the card), + which is charged back out of the cache whenever the cache lends (`vram_budget.cpp:193-203`). Boots of + evidence, all on 1.8.1: + + | shape | demand | cache given | engine's `backs %` | |---|---|---|---| - | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) | + | 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % | | 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % | - | 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) | + | 2 × 200,000, host pool 44,928 | 400,000 tok | 2.76 GiB | 91 % | + | 4 × 200,000, host pool 49,152 (live) | 800,000 tok | **5.64 GiB = 747,492 tok** | 93 % | - So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler — - never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a - slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is - grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle. - `--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot - hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8 - era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` — - the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over - `--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache. + The 45,568 row is the ceiling case, not a mystery: at 3 slots the cache already held its whole + `ceiling`, so extra host room could not grow it and shows up as `unclaimed`. **The referee is the + ledger line `kv: the pool backs N % … the cache can grow to … N tokens`, never arithmetic.** To buy + window: raise `--host-pool-mib` (cost: pinned host RAM, and experts evicted to host cross PCIe — + `experts-from-host` was 5.99 GiB a rank at 44,928 and is 8.99 at 49,152) or cut the per-token cost + (`fp8` KV). Raising slots raises demand **and** costs 10 × 111.98 MiB of pinned checkpoint-store RAM + per slot. Under-sizing the pool the other way has always been a boot failure (36,864 refused a boot in + the fp8 era: the host cannot hold what the cards will not). Behavioural check on the live shape: + `/health` reports `queue_cap 32`; a request beyond the slot count queues rather than being refused, + and the 400 class is a single request over `--max-model-len`. - **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275 slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era) @@ -366,17 +389,18 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). `--checkpoint-interval` does **not** buy the deep case: the coarsest ladder scale is the largest `interval × 2^k` under the depth, so the deepest gap stays ≈ half the prompt (129,024 tokens at interval 2048 against 130,816 at 256) while every cut multiplies the snapshot writes. -- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it - adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly), - declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows - (single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6 - today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % - floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it - moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while - the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger - mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the - retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first - fp8-native start. +- **Slots cost KV, host RAM, rows, and pin — never speed.** `--max-num-seqs` carries no radiance + constraint: it adds ≈110 MiB of recurrent state per slot, 10 × 111.98 MiB of pinned checkpoint-store + RAM per slot (`--checkpoint-slots auto`), and declares the fused TP all-reduce over + `seqs × (1 + spec)` rows — `wht6` is bit-exact up to 25 rows, and the live shape is 4 × 3 = **12**, so + single-sequence decode stays exact and so does any step up to that many rows. Raising it also raises + the cache's *demand* (`ceiling = slots × window`), which the resolver funds from `elastic − + expert_floor` — so a slot count without matching host room is paid for by expert residency. Measured + 2026-10-06: seqs 2 and 3 statistically tied (30.9-33.1 t/s, inside the ±3 % floor). Measured + 2026-10-19 → 10-20 at 1.8.1: 2 → 4 slots cost **+4,224 MiB of pinned host pool** and **−3.01 GiB of + expert residency a card** and bought 2.76 → 5.64 GiB of cache (365,204 → 747,492 tokens). The old + "seqs 2 cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape + specialization, and radiance booted seqs 2 on its first fp8-native start. - **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the weight count. Read the engine log for @@ -570,13 +594,13 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`. - **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request longer than the KV pool's token count is accepted then refused as too large to fit; a request - beyond `--max-num-seqs` is held by the HTTP port (`queue_cap`, default 8 × slots = 16 here) and - answered 503 only past that, never paged or squeezed. Because the cache funds **1.8** full-height - sessions, admitting a second near-full-height conversation can **preempt a running sequence from - the back** to free its blocks (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so - its session halts and then re-prefills. Meter: `vllm:num_preemptions_total` (264 in the current - boot, all in prefill-coincident bursts — 108 inside 105 s); `/stats preemptions` is the cumulative - count and `radiance:num_requests_preempted` only the instant depth. + beyond `--max-num-seqs` is held by the HTTP port (`queue_cap` = `max(8, 8 × slots)` — **32** here) + and answered 503 only past that, never paged or squeezed. The cache funds **3.7** full-height + sessions, so four conversations can be resident but a fourth near-full-height one can still + **preempt a running sequence from the back** to free its blocks + (`core/sched/scheduler.cpp:1291-1322`); the victim is recomputed, so its session halts and then + re-prefills. Meter: `vllm:num_preemptions_total` (cumulative for the boot, `/stats preemptions`); + `radiance:num_requests_preempted` is only the instant depth. - **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the Dankclaw Prometheus job at `192.168.0.235:7770` (15 s interval) over plain LAN routing (the netavark published-port quirk is same-host only). **Latency lives under `vllm:`, cadence under