feat(frank-archon): arm --prefill-interleave at 1

A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
This commit is contained in:
Frank committed 2026-10-20 22:52:31 -07:00
1 parent b1fc1113c6
commit aa75b45543
2 files changed
+12 -6

No files matched your search

+8 -2
View File
@@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# 512 is a trap and the vendor measures the whole job slower there. Interleave # 512 is a trap and the vendor measures the whole job slower there. Interleave
# inserts N decode-only steps after a chunk step taken beside decoders: it protects # inserts N decode-only steps after a chunk step taken beside decoders: it protects
# their cadence and holds a waiting request out of those steps. That is the decoder # their cadence and holds a waiting request out of those steps. That is the decoder
# dial, and it is 0 until the decode-vs-prefill buckets ask for it. # dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
# price falls on the prefilling prompt (its chunks lose every held step) and on any
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
# ALONE, so it cannot buy this back while decoders run. Meter:
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
# before N rises.
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
FRANK_ARCHON_PREFILL_INTERLEAVE=0 FRANK_ARCHON_PREFILL_INTERLEAVE=1
# Prefix cache: pinned host memory only. The disk tier was removed outright # Prefix cache: pinned host memory only. The disk tier was removed outright
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to # (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one # be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
+4 -4
View File
@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) | | Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache | | Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it | | Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it | | Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` | | TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) | | Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
@@ -327,11 +327,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
says the binding constraint in one line: "a prefill step streams every routed layer's pooled says the binding constraint in one line: "a prefill step streams every routed layer's pooled
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to". experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing. Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).** - **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33, two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not ~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
substitute for it. substitute for it.
- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N - **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N