feat(frank-archon): move the endpoint to radiance 1.8.1 and its budget model
The decode stall is step contention (measured: 18.7 ms decode-only steps
against 317 ms steps that carry a prefill chunk, decode at 13 t/s median
while prefill runs), and the two engine knobs that fix it arrived after this
endpoint's pin. Moving to 1.8.1 (2026-10-18) brings both, and brings two CLI
breaks with them:
- --vram-weights-mib / --vram-kv-mib were REMOVED at 1.3.0. The per-card
split between the expert slab and the KV pool is now derived by the
engine's budget resolver from --gpu-headroom-mib, the cache bound
(--max-model-len, --max-num-seqs) and --host-pool-mib, and the cache may
lend idle blocks to the expert slab. The slab size and the pool's token
count are boot-ledger lines from here on, so the capacity contract
(3 x 200,000 fp8) is verified live by the probe's capacity leg instead of
computed from an env number.
- The prefix cache has no disk tier any more: --prefix-cache-dir and
--prefix-cache-disk-mib are refusals and --prefix-cache-host-mib is
--prefix-cache-mib. 64 GiB of parked KV on NVMe becomes re-prefill, so
prefix_hit_rate is the thing to watch after this boot.
env: image re-pinned to the 1.8.1 digest; the pair and the disk tier dropped;
FRANK_ARCHON_PREFIX_CACHE_MIB replaces the host/disk pair; new
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto (explicit, because auto vs off is the
difference between decode waiting one chunk and waiting a whole step) and
FRANK_ARCHON_PREFILL_INTERLEAVE=0 (opt-in through a guarded argv array).
serve: argv follows the new CLI, /data mount and the pre-created kvcache path
go with the disk tier, and the retired env keys are no longer required (they
would abort ExecStart). autoresearch-frank-archon.sh takes the same CLI, with
a cold prefill now coming from --prefix-cache-mib 0 instead of a per-run disk
directory. run-swift15-bf16.sh is untouched: it pins its own engines by design.
tests: the two budget-arithmetic seams asserted knobs that no longer exist and
are replaced by one seam that asserts the retired knobs are ABSENT from env and
argv - a stale line is a boot refusal, not a no-op. The container-contract seam
now asserts the tune mount and the absence of /data.
This commit is contained in:
1 parent
b9d21ae947
commit
9de58ed6fd
4 files changed
+117
-118
No files matched your search
+42
-19
@@ -7,10 +7,15 @@
|
||||
# only the reason a reader editing THIS line cannot recover from the page.
|
||||
# Where the two disagree, this file wins on values; the boot ledger arbitrates
|
||||
# the engine shape.
|
||||
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||
# 1.0.13 is pinned by digest on purpose: 1.2.0 is a DEAD END on this payload
|
||||
# (acceptance 0.7123 vs 0.7549 — the page's Dead ends section names the suspect).
|
||||
# The .rad file mounted under /models; /data holds the prefix-cache disk tier.
|
||||
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed
|
||||
# 1.8.1 (2026-10-18), pinned by digest. Two CLI breaks sit between it and the
|
||||
# 1.0.13 line this endpoint ran for two eras, and both are boot refusals, not
|
||||
# warnings: --vram-weights-mib/--vram-kv-mib were REMOVED at 1.3.0 (the card is
|
||||
# now split by the engine's budget resolver — see below), and the prefix cache
|
||||
# lost its disk tier. Draft acceptance has to be re-scored on any pin past 1.2.0:
|
||||
# 1.2.0's MoE weight rotation measured 0.7123 against the 0.7549 control, and the
|
||||
# suspect is untangled (the page's Dead ends section names it).
|
||||
# The .rad file mounts under /models; the tune cache mounts separately below.
|
||||
# The model dir holds ONLY .rad files — the HF download and staging stay beside it.
|
||||
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
|
||||
@@ -52,21 +57,25 @@ FRANK_ARCHON_MAX_SEQS=3
|
||||
# sequence.
|
||||
FRANK_ARCHON_MAX_MODEL_LEN=200000
|
||||
FRANK_ARCHON_PLACEMENT=expert_tiered
|
||||
# Pinned host pool for expert rows that do not fit VRAM. It moves WITH
|
||||
# VRAM_WEIGHTS: the pair and this pool serve the per-rank resident set together,
|
||||
# so cutting the slab spills here — under-sizing it is a boot failure (36,864
|
||||
# Pinned host pool for expert rows that do not fit VRAM. Under the budget
|
||||
# resolver this is ALSO the lever that decides which experts stay on the card:
|
||||
# the card's leftover goes to the expert plane first and the KV cache takes the
|
||||
# rest, so growing the pool moves experts off the card and shrinking it spills
|
||||
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
|
||||
# refused a boot). The ledger's coverage line is the referee, not this file.
|
||||
FRANK_ARCHON_HOST_POOL_MIB=44928
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
||||
# floor is built from it.
|
||||
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||
# ONE budget per card: VRAM_WEIGHTS + VRAM_KV <= 30,969 MiB, the ceiling the
|
||||
# engine's own budget table prints (derivation and the slab/pool price: the
|
||||
# page's VRAM laws). Idle KV VRAM is ALSO the expert cache's spillover, so a
|
||||
# bigger pool pays twice — which is why the pool sits at the contract and the
|
||||
# slab takes the rest.
|
||||
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
|
||||
FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# There is no VRAM pair any more. --vram-weights-mib/--vram-kv-mib were removed
|
||||
# at 1.3.0: the engine takes capacity - already-held - GPU_HEADROOM - arena,
|
||||
# puts the static (non-expert) weights off the top, then divides the rest between
|
||||
# the expert plane and the KV cache — experts first when the cache has no idle
|
||||
# blocks to lend, and never less than ONE MAX_MODEL_LEN sequence for the cache.
|
||||
# What moves the split from here is HOST_POOL_MIB (which experts stay resident),
|
||||
# MAX_MODEL_LEN/MAX_SEQS (the cache bound) and GPU_HEADROOM_MIB. The slab size
|
||||
# the prefill-slope law cares about is a boot-ledger line now, not a knob: read
|
||||
# it after every resize instead of asserting it here.
|
||||
# fp8 is the ONLY rung below bf16 on this engine: one kv_dtype for the whole
|
||||
# cache, so llama.cpp's exact-K/quantized-V split (frank-zealot) is unservable
|
||||
# here. bf16 costs 1.89x fp8, not 2x — the QSA block-key plane is dtype-fixed.
|
||||
@@ -75,11 +84,25 @@ FRANK_ARCHON_KV_DTYPE=fp8
|
||||
# 2, and 3 x (1+2) = 9 rows stays inside the wht6 25-row exactness bound.
|
||||
FRANK_ARCHON_SPEC_TOKENS=2
|
||||
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# Prefix cache tiers. The host tier is sized in ENTRIES and one entry tracks the
|
||||
# cache dtype (58.50 KiB at fp8), so a dtype change re-sizes the parked set
|
||||
# without touching these numbers.
|
||||
FRANK_ARCHON_PREFIX_HOST_MIB=8192
|
||||
FRANK_ARCHON_PREFIX_DISK_MIB=65536
|
||||
# Step sharing. A prefill chunk that shares a step with decoding sequences holds
|
||||
# the device for the chunk's whole time, and every decode row in that step waits
|
||||
# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
|
||||
# `auto` leaves each decode row behind the chunk a row plus its draft window and
|
||||
# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
|
||||
# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
|
||||
# chunk took the step and admission never ran. Interleave inserts N decode-only
|
||||
# steps after a chunk step taken beside decoders: it protects their cadence and
|
||||
# holds a waiting request out of those steps, so it stays 0 until the post-migration
|
||||
# decode-vs-prefill buckets say it is wanted.
|
||||
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
|
||||
FRANK_ARCHON_PREFILL_INTERLEAVE=0
|
||||
# Prefix cache: pinned host memory only. The disk tier was removed outright
|
||||
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
|
||||
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
|
||||
# entry tracks the cache dtype (58.50 KiB at fp8). Watch prefix_hit_rate after
|
||||
# this change: a drop is extra prefill, and prefill is exactly the traffic that
|
||||
# costs decode its cadence.
|
||||
FRANK_ARCHON_PREFIX_CACHE_MIB=8192
|
||||
# Frank's pin law: the PLE n-gram table serves from pinned host RAM the cards
|
||||
# read. `ram` REFUSES the start if memlock is too small — the unit's
|
||||
# LimitMEMLOCK=infinity is the other half of this law.
|
||||
|
||||
Reference in new issue
Block a user