feat(frank-archon): arm --prefill-interleave at 1

A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
This commit is contained in:
Frank
2026-10-20 22:52:31 -07:00
parent b1fc1113c6
commit aa75b45543
2 changed files with 12 additions and 6 deletions
+8 -2
View File
@@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# 512 is a trap and the vendor measures the whole job slower there. Interleave
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
# their cadence and holds a waiting request out of those steps. That is the decoder
# dial, and it is 0 until the decode-vs-prefill buckets ask for it.
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
# price falls on the prefilling prompt (its chunks lose every held step) and on any
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
# ALONE, so it cannot buy this back while decoders run. Meter:
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
# before N rises.
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
FRANK_ARCHON_PREFILL_INTERLEAVE=0
FRANK_ARCHON_PREFILL_INTERLEAVE=1
# Prefix cache: pinned host memory only. The disk tier was removed outright
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
+4 -4
View File
@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it |
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
@@ -327,11 +327,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
says the binding constraint in one line: "a prefill step streams every routed layer's pooled
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).**
- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33,
N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
substitute for it.
- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N