diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 65ac22c..3adb083 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048 # 512 is a trap and the vendor measures the whole job slower there. Interleave # inserts N decode-only steps after a chunk step taken beside decoders: it protects # their cadence and holds a waiting request out of those steps. That is the decoder -# dial, and it is 0 until the decode-vs-prefill buckets ask for it. +# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step +# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the +# price falls on the prefilling prompt (its chunks lose every held step) and on any +# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is +# ALONE, so it cannot buy this back while decoders run. Meter: +# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move +# before N rises. FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto -FRANK_ARCHON_PREFILL_INTERLEAVE=0 +FRANK_ARCHON_PREFILL_INTERLEAVE=1 # Prefix cache: pinned host memory only. The disk tier was removed outright # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 855c2e7..9027e47 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | -| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it | +| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | @@ -327,11 +327,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). says the binding constraint in one line: "a prefill step streams every routed layer's pooled experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to". Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing. -- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).** +- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.** N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the - two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33, - N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not + two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) → + ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a substitute for it. - **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N