feat(frank-archon): arm --prefill-interleave at 1
A prefill chunk that shares a step with decoding sequences holds the device for the chunk's whole time: measured on this shape, decode-only steps 15.5 ms against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for an arriving request goes 2.4 -> 18.0 s. BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source refuses to halve the chunk for decode rows alone -- which is why engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE is the knob that acts on the measured mechanism: N decode-only steps follow any chunk step taken beside decoders. N=1 is the conservative first arm -- a stream takes its token from the chunk step and one from the held step (~2 tokens per 182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's TTFT and on requests still queued, not on anything decoding. Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the serve script only passes it when > 0), engine_steps_held_total must leave zero, then the §9 bucket pair must move before N rises.
This commit is contained in:
@@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# 512 is a trap and the vendor measures the whole job slower there. Interleave
|
||||
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
|
||||
# their cadence and holds a waiting request out of those steps. That is the decoder
|
||||
# dial, and it is 0 until the decode-vs-prefill buckets ask for it.
|
||||
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
|
||||
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
|
||||
# price falls on the prefilling prompt (its chunks lose every held step) and on any
|
||||
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
|
||||
# ALONE, so it cannot buy this back while decoders run. Meter:
|
||||
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
|
||||
# before N rises.
|
||||
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
|
||||
FRANK_ARCHON_PREFILL_INTERLEAVE=0
|
||||
FRANK_ARCHON_PREFILL_INTERLEAVE=1
|
||||
# Prefix cache: pinned host memory only. The disk tier was removed outright
|
||||
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
|
||||
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
|
||||
|
||||
@@ -96,7 +96,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
|
||||
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
|
||||
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
|
||||
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
||||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=0` — the decoder dial, held at zero until the §9 buckets ask for it |
|
||||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` — prefill-speed first: it reserves 3 tokens per waiting decoder and ⅛ of the budget per waiting request, and does **not** shrink the chunk for decoders alone · `PREFILL_INTERLEAVE=1` — the decoder dial, **armed 2026-10-20**: one decode-only step after each chunk step taken beside decoders; meter `engine_steps_held_total`, price is the prefilling prompt's TTFT |
|
||||
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
||||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||||
|
||||
@@ -327,11 +327,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
says the binding constraint in one line: "a prefill step streams every routed layer's pooled
|
||||
experts onto the card, so its speed is bounded by the link; decode reads the ones it routes to".
|
||||
Measured size of the whole effect: the ~10 % above. Real, small, not worth chasing.
|
||||
- **`--prefill-interleave` is the dial that buys decode its cadence; it is off (`steps_held` 0).**
|
||||
- **`--prefill-interleave` is the dial that buys decode its cadence; armed at 1 on 2026-10-20.**
|
||||
N decode-only steps follow any chunk step taken beside decoders; held steps admit nothing, so it
|
||||
costs the prefilling prompt its TTFT and an arriving request its first step. Rate arithmetic at the
|
||||
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, N=2 → ~33,
|
||||
N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
|
||||
two measured step times and 0.657 acceptance (≈2.3 tokens a step): N=0 → ~14 t/s, **N=1 (armed) →
|
||||
~23**, N=2 → ~33, N=4 → ~48, N=8 → ~69, with the prefill slowed by (166 + 15.5N)/166. **Arithmetic, not
|
||||
measurement** — the arm is chosen by the §9 bucket pair plus `steps_held`, and `auto` is not a
|
||||
substitute for it.
|
||||
- **`--prefill-span` is the shape that keeps the loan, and it is priced.** It needs MNBT = N
|
||||
|
||||
Reference in New Issue
Block a user