dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-archon): arm --prefill-interleave at 1

Browse Source 
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
This commit is contained in:
Frank
2026-10-20 22:52:31 -07:00
parent b1fc1113c6
commit aa75b45543
2 changed files with 12 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-archon.env
+8 -2
View File
@@ -107,9 +107,15 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# 512 is a trap and the vendor measures the whole job slower there. Interleave
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
# their cadence and holds a waiting request out of those steps. That is the decoder
# dial, and it is 0 until the decode-vs-prefill buckets ask for it.
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
# price falls on the prefilling prompt (its chunks lose every held step) and on any
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
# ALONE, so it cannot buy this back while decoders run. Meter:
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
# before N rises.
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
FRANK_ARCHON_PREFILL_INTERLEAVE=0
FRANK_ARCHON_PREFILL_INTERLEAVE=1
# Prefix cache: pinned host memory only. The disk tier was removed outright
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one