FRANK_ARCHON_PREFILL_INTERLEAVE=0 → 1 on frank-archon. Nothing else in the trio moves: same image pin, same payload, same BUSY_PREFILL_CHUNK=auto, same slots and window. The serve script passes --prefill-interleave only when the value is > 0, so this adds exactly one argv flag.
Why
Census on the live 1.8.1 shape (48 h of Prometheus + /stats, method in LLM Performance Diagnosis §9):
quantity
decode-only step
step carrying a prefill chunk
mean step time
15.5 ms
166 ms
share of engine step time
66 % of steps, 66 % of time
4.7 % of steps, 34 % of time
harness-visible decode
166 tok/s (light prefill)
13 tok/s (> 5k prefill tok/s) — 6.7 tok/s per stream at 2 sessions
Decode-only steps stay 14–18 ms in every prefill bucket and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from 30 s on), so eviction/expert-residency is not the mechanism: decode is paying for the step it shares with a chunk, one token per 166 ms. The second victim is the arriving request — TTFT p90 2.4 → 18.0 s and 15.5 % of requests over 10 s while prefill runs high.
BUSY_PREFILL_CHUNK=auto is not the protection: the pinned source (core/sched/scheduler.cppbusy_cap) reserves 3 tokens per decoding sequence and ⅛ of the budget per waiting request and refuses to halve the chunk for decode rows alone. The proof is engine_steps_held_total reading zero for the whole 34.7 h boot — nothing was protecting anyone.
--prefill-interleave N is the knob that acts on the measured mechanism: after a chunk step taken beside decoders, N steps carry decode rows only and admit nothing. N=1 is the conservative rung — a stream takes its token from the chunk step and one from the held step, ~2 tokens per 182 ms instead of one per 166 ms. The cost lands on the prefilling prompt's TTFT and on queued requests, never on a session already decoding.
Verification
./tests/run.sh — 472 tests, green.
Post-deploy: --prefill-interleave 1 present in /proc/$(pidof radiance)/cmdline; /health 200; one chat turn, one tool call, one vision probe (this endpoint takes images natively).
The arm's own meter: radiance:engine_steps_held_total must leave zero. Then the §9 bucket pair (decode-only vs chunk step time, and the decode-by-prefill buckets) decides whether N rises. The rate ladder is arithmetic, not measurement: N=1 → ~23 t/s, N=2 → ~33, N=8 → ~69, prefill slowed by (166 + 15.5N)/166.
Rollback
FRANK_ARCHON_PREFILL_INTERLEAVE=0, redeploy --service frank-archon. One line, no rebuild.
Also in this branch (docs only)
687dbfb the env/serve comments that claimed auto protects decoders, and the serve header still advertising three decode slots.
9f7457e the interference census written as laws on frank-archon: latency metrics live under vllm: and cadence under radiance:; preemption is vllm:num_preemptions_total (264 this boot, 108 inside 105 s) while the radiance gauge is instant depth; both GPU root ports are x8 Gen5 max by board; DDR is back at 4800 MT/s while the recorded corruption fix was 4600.
b1fc111 destroys an invented flag name (--checkpoint-policy floor) and records the three that exist, priced from core/mem/prefix.{h,cpp} — current setting is the default geometric-backoff, 20 slots / 2.19 GiB pinned, and keep-all would ask 196 slots / 21.4 GiB.
## What changes
`FRANK_ARCHON_PREFILL_INTERLEAVE=0 → 1` on `frank-archon`. Nothing else in the trio moves: same image pin, same payload, same `BUSY_PREFILL_CHUNK=auto`, same slots and window. The serve script passes `--prefill-interleave` only when the value is > 0, so this adds exactly one argv flag.
## Why
Census on the live 1.8.1 shape (48 h of Prometheus + `/stats`, method in [[LLM Performance Diagnosis]] §9):
| quantity | decode-only step | step carrying a prefill chunk |
|---|---|---|
| mean step time | **15.5 ms** | **166 ms** |
| share of engine step time | 66 % of steps, 66 % of time | **4.7 % of steps, 34 % of time** |
| harness-visible decode | **166 tok/s** (light prefill) | **13 tok/s** (> 5k prefill tok/s) — 6.7 tok/s per stream at 2 sessions |
Decode-only steps stay 14–18 ms in every prefill bucket and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from 30 s on), so eviction/expert-residency is not the mechanism: **decode is paying for the step it shares with a chunk**, one token per 166 ms. The second victim is the arriving request — TTFT p90 2.4 → 18.0 s and 15.5 % of requests over 10 s while prefill runs high.
`BUSY_PREFILL_CHUNK=auto` is not the protection: the pinned source (`core/sched/scheduler.cpp` `busy_cap`) reserves 3 tokens per decoding sequence and ⅛ of the budget per waiting request and refuses to halve the chunk for decode rows alone. The proof is `engine_steps_held_total` reading zero for the whole 34.7 h boot — nothing was protecting anyone.
`--prefill-interleave N` is the knob that acts on the measured mechanism: after a chunk step taken beside decoders, N steps carry decode rows only and admit nothing. N=1 is the conservative rung — a stream takes its token from the chunk step **and** one from the held step, ~2 tokens per 182 ms instead of one per 166 ms. The cost lands on the prefilling prompt's TTFT and on queued requests, never on a session already decoding.
## Verification
- `./tests/run.sh` — 472 tests, green.
- Post-deploy: `--prefill-interleave 1` present in `/proc/$(pidof radiance)/cmdline`; `/health` 200; one chat turn, one tool call, one vision probe (this endpoint takes images natively).
- The arm's own meter: `radiance:engine_steps_held_total` must leave zero. Then the §9 bucket pair (decode-only vs chunk step time, and the decode-by-prefill buckets) decides whether N rises. The rate ladder is arithmetic, not measurement: N=1 → ~23 t/s, N=2 → ~33, N=8 → ~69, prefill slowed by (166 + 15.5N)/166.
## Rollback
`FRANK_ARCHON_PREFILL_INTERLEAVE=0`, redeploy `--service frank-archon`. One line, no rebuild.
## Also in this branch (docs only)
- `687dbfb` the env/serve comments that claimed `auto` protects decoders, and the serve header still advertising three decode slots.
- `9f7457e` the interference census written as laws on [[frank-archon]]: latency metrics live under `vllm:` and cadence under `radiance:`; preemption is `vllm:num_preemptions_total` (264 this boot, 108 inside 105 s) while the radiance gauge is instant depth; both GPU root ports are x8 Gen5 max by board; DDR is back at 4800 MT/s while the recorded corruption fix was 4600.
- `b1fc111` destroys an invented flag name (`--checkpoint-policy floor`) and records the three that exist, priced from `core/mem/prefix.{h,cpp}` — current setting is the default `geometric-backoff`, 20 slots / 2.19 GiB pinned, and `keep-all` would ask 196 slots / 21.4 GiB.
---
cwd: /home/blank/.omp/wt/wt-20261020-213526-6d1fc1f · session: 01a15cbe-157e-7527-b4b5-22f3d2feea00
The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1
source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding
sequence and 1/8 of the budget per waiting request, and refuses to shrink the
chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment
carried a 512 that is not this build's auto value (0 until the held-step
counter asks; 64 in help mode). The serve header still described three decode
slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.
frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.
Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.
§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly
geometric-backoff (default), tip-only, keep-all and refuses anything else at
boot, so the recorded arm could never be deployed. Replaced with the real
semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric
ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live
ledger line -- so a turn diverging d behind the tip can resume at tip-3d:
diverge at 120k, replay 21.7k); retention is not the write path, so keep-all
buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4
GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room);
and shortening --checkpoint-interval cannot fix the deep case -- the coarsest
ladder scale is interval x 2^k under the depth, so the deepest gap stays about
half the prompt at every interval. §9 keeps only the three-name locator.
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.
BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.
Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
dank
merged commit a4edaa7d9d into main2026-10-20 22:55:58 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What changes
FRANK_ARCHON_PREFILL_INTERLEAVE=0 → 1onfrank-archon. Nothing else in the trio moves: same image pin, same payload, sameBUSY_PREFILL_CHUNK=auto, same slots and window. The serve script passes--prefill-interleaveonly when the value is > 0, so this adds exactly one argv flag.Why
Census on the live 1.8.1 shape (48 h of Prometheus +
/stats, method in LLM Performance Diagnosis §9):Decode-only steps stay 14–18 ms in every prefill bucket and the relationship is lag-0 only (r = −0.74 at lag 0, |r| ≤ 0.19 from 30 s on), so eviction/expert-residency is not the mechanism: decode is paying for the step it shares with a chunk, one token per 166 ms. The second victim is the arriving request — TTFT p90 2.4 → 18.0 s and 15.5 % of requests over 10 s while prefill runs high.
BUSY_PREFILL_CHUNK=autois not the protection: the pinned source (core/sched/scheduler.cppbusy_cap) reserves 3 tokens per decoding sequence and ⅛ of the budget per waiting request and refuses to halve the chunk for decode rows alone. The proof isengine_steps_held_totalreading zero for the whole 34.7 h boot — nothing was protecting anyone.--prefill-interleave Nis the knob that acts on the measured mechanism: after a chunk step taken beside decoders, N steps carry decode rows only and admit nothing. N=1 is the conservative rung — a stream takes its token from the chunk step and one from the held step, ~2 tokens per 182 ms instead of one per 166 ms. The cost lands on the prefilling prompt's TTFT and on queued requests, never on a session already decoding.Verification
./tests/run.sh— 472 tests, green.--prefill-interleave 1present in/proc/$(pidof radiance)/cmdline;/health200; one chat turn, one tool call, one vision probe (this endpoint takes images natively).radiance:engine_steps_held_totalmust leave zero. Then the §9 bucket pair (decode-only vs chunk step time, and the decode-by-prefill buckets) decides whether N rises. The rate ladder is arithmetic, not measurement: N=1 → ~23 t/s, N=2 → ~33, N=8 → ~69, prefill slowed by (166 + 15.5N)/166.Rollback
FRANK_ARCHON_PREFILL_INTERLEAVE=0, redeploy--service frank-archon. One line, no rebuild.Also in this branch (docs only)
687dbfbthe env/serve comments that claimedautoprotects decoders, and the serve header still advertising three decode slots.9f7457ethe interference census written as laws on frank-archon: latency metrics live undervllm:and cadence underradiance:; preemption isvllm:num_preemptions_total(264 this boot, 108 inside 105 s) while the radiance gauge is instant depth; both GPU root ports are x8 Gen5 max by board; DDR is back at 4800 MT/s while the recorded corruption fix was 4600.b1fc111destroys an invented flag name (--checkpoint-policy floor) and records the three that exist, priced fromcore/mem/prefix.{h,cpp}— current setting is the defaultgeometric-backoff, 20 slots / 2.19 GiB pinned, andkeep-allwould ask 196 slots / 21.4 GiB.cwd: /home/blank/.omp/wt/wt-20261020-213526-6d1fc1f · session: 01a15cbe-157e-7527-b4b5-22f3d2feea00
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly geometric-backoff (default), tip-only, keep-all and refuses anything else at boot, so the recorded arm could never be deployed. Replaced with the real semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live ledger line -- so a turn diverging d behind the tip can resume at tip-3d: diverge at 120k, replay 21.7k); retention is not the write path, so keep-all buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4 GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room); and shortening --checkpoint-interval cannot fix the deep case -- the coarsest ladder scale is interval x 2^k under the depth, so the deepest gap stays about half the prompt at every interval. §9 keeps only the three-name locator.