Frank
d5934a6b47
The 1.8.1 boot resolved the budget resolver to 4.20 GiB of cache a card — 556,348 tokens, 2.8 sessions at full height — so three concurrent full-height requests are refused with a 400 at admission: the endpoint's 3-slot contract is broken by the cutover, silently, because /health and every shorter conversation still answer. The resolver gives the expert plane whatever the pinned host pool cannot hold and the cache the rest, so the pool is the lever: +640 MiB a rank moves that much expert demand into host RAM and the cache wins it back. Tokens scale with the GiB won (8.11 KiB/token at fp8), so ~0.63 GiB buys ~44,000 tokens — 600,000 with margin — at ~2.5% off the expert slab. TTM floor moves with the pool (2 x pool + 48,824 + prefix + MNBT = 150,200 MiB computed by the ledger test). Verified by the boot ledger's `kv:` line and the probe's capacity leg, not by this arithmetic.
131 lines
8.2 KiB
Bash
131 lines
8.2 KiB
Bash
# frank-archon.env — radiance (StillDeadcode) on BOTH R9700s (TP=2),
|
|
# Dankserver :7770, publisher's 4-bit-codebook container of the official
|
|
# Qwen3.8-Flash-Next.
|
|
#
|
|
# ONE OWNER: shape, payload identity, eras, laws and every measured number live
|
|
# in serverAppConfigs-docs/LLM/frank-archon.md. This file carries the knobs plus
|
|
# only the reason a reader editing THIS line cannot recover from the page.
|
|
# Where the two disagree, this file wins on values; the boot ledger arbitrates
|
|
# the engine shape.
|
|
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed
|
|
# 1.8.1 (2026-10-18), pinned by digest. Two CLI breaks sit between it and the
|
|
# 1.0.13 line this endpoint ran for two eras, and both are boot refusals, not
|
|
# warnings: --vram-weights-mib/--vram-kv-mib were REMOVED at 1.3.0 (the card is
|
|
# now split by the engine's budget resolver — see below), and the prefix cache
|
|
# lost its disk tier. Draft acceptance has to be re-scored on any pin past 1.2.0:
|
|
# 1.2.0's MoE weight rotation measured 0.7123 against the 0.7549 control, and the
|
|
# suspect is untangled (the page's Dead ends section names it).
|
|
# The .rad file mounts under /models; the tune cache mounts separately below.
|
|
# The model dir holds ONLY .rad files — the HF download and staging stay beside it.
|
|
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
|
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
|
|
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
|
|
# state dir's tune/ subdir carries rad-tune's kernel-selection cache; the engine
|
|
# reads $RADIANCE_HOME/tune/<machine>.tune and that defaults to /opt/radiance
|
|
# (inside the image), so the cache only survives on a host mount.
|
|
# Container identity is the SERVICE name: one name across systemctl and
|
|
# `podman ps`; the runtime changes, the name does not. The legacy list retires
|
|
# every predecessor identity that could hold the GPUs or the port.
|
|
FRANK_ARCHON_CONTAINER=frank-archon
|
|
FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
|
|
# Served id. Radiance VALIDATES the requested model name, so every client must
|
|
# send exactly this value (unlike llama.cpp and halogen, which ignore it).
|
|
FRANK_ARCHON_SERVED_NAME=frank-archon
|
|
FRANK_ARCHON_PORT=7770
|
|
# Frank's chat-template call: froggeric's fixed Qwen3.8 template, vendored in
|
|
# the repo; it overrides the template baked into the .rad, and radiance derives
|
|
# the reasoning and tool-call reply format from it.
|
|
FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
|
|
|
# --- engine shape (the boot ledger arbitrates every number below) ---------
|
|
# TP=2 needs the 2026-09-20 board (x8/x8 on root ports): a collective cannot be
|
|
# dispatched on a card whose PCIe chain lacks atomics. Radiance's wire is
|
|
# libr4d's own, not RCCL.
|
|
FRANK_ARCHON_TP=2
|
|
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice). It
|
|
# stays EXACT for single-sequence decode and any step of <= 25 rows — see
|
|
# MAX_SEQS x (1 + SPEC_TOKENS) below.
|
|
FRANK_ARCHON_TP_WIRE=wht6
|
|
# Decode slots. A slot costs KV, never decode; the pool is sized to this
|
|
# contract and nothing more, so a FOURTH full-height conversation is REFUSED as
|
|
# too large to fit — never squeezed, paged or truncated. That is Frank's
|
|
# explicit trade: the freed MiB went to the expert slab.
|
|
FRANK_ARCHON_MAX_SEQS=3
|
|
# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
|
|
# limit, and models.yml contextWindow MUST quote this number (over-length is a
|
|
# refusal, not a truncation). The pool is sized for seqs x maxlen, not one
|
|
# sequence.
|
|
FRANK_ARCHON_MAX_MODEL_LEN=200000
|
|
FRANK_ARCHON_PLACEMENT=expert_tiered
|
|
# Pinned host pool for expert rows that do not fit VRAM. Under the budget
|
|
# resolver this is ALSO the lever that decides which experts stay on the card:
|
|
# the card's leftover goes to the expert plane first and the KV cache takes the
|
|
# rest, so growing the pool moves experts off the card and shrinking it spills
|
|
# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
|
|
# refused a boot). The ledger's coverage line is the referee, not this file.
|
|
# Sized so the cache clears the 3 x MAX_MODEL_LEN contract: the 1.8.1 boot at
|
|
# 44,928 resolved the cache to 556,348 tokens — 2.8 sessions — and three
|
|
# concurrent full-height requests came back 400 in 0.0 s. Tokens scale with the
|
|
# GiB the cache wins, so +640 MiB a rank buys ~44,000 tokens of ceiling with
|
|
# ~2.5% off the expert slab.
|
|
FRANK_ARCHON_HOST_POOL_MIB=45568
|
|
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
|
|
# floor is built from it.
|
|
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
|
# There is no VRAM pair any more. --vram-weights-mib/--vram-kv-mib were removed
|
|
# at 1.3.0: the engine takes capacity - already-held - GPU_HEADROOM - arena,
|
|
# puts the static (non-expert) weights off the top, then divides the rest between
|
|
# the expert plane and the KV cache — experts first when the cache has no idle
|
|
# blocks to lend, and never less than ONE MAX_MODEL_LEN sequence for the cache.
|
|
# What moves the split from here is HOST_POOL_MIB (which experts stay resident),
|
|
# MAX_MODEL_LEN/MAX_SEQS (the cache bound) and GPU_HEADROOM_MIB. The slab size
|
|
# the prefill-slope law cares about is a boot-ledger line now, not a knob: read
|
|
# it after every resize instead of asserting it here.
|
|
# fp8 is the ONLY rung below bf16 on this engine: one kv_dtype for the whole
|
|
# cache, so llama.cpp's exact-K/quantized-V split (frank-zealot) is unservable
|
|
# here. bf16 costs 1.89x fp8, not 2x — the QSA block-key plane is dtype-fixed.
|
|
FRANK_ARCHON_KV_DTYPE=fp8
|
|
# The checkpoint's own MTP head at depth 2: this fleet's ladder was measured at
|
|
# 2, and 3 x (1+2) = 9 rows stays inside the wht6 25-row exactness bound.
|
|
FRANK_ARCHON_SPEC_TOKENS=2
|
|
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
|
# Step sharing. A prefill chunk that shares a step with decoding sequences holds
|
|
# the device for the chunk's whole time, and every decode row in that step waits
|
|
# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
|
|
# `auto` leaves each decode row behind the chunk a row plus its draft window and
|
|
# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
|
|
# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
|
|
# chunk took the step and admission never ran. Interleave inserts N decode-only
|
|
# steps after a chunk step taken beside decoders: it protects their cadence and
|
|
# holds a waiting request out of those steps, so it stays 0 until the post-migration
|
|
# decode-vs-prefill buckets say it is wanted.
|
|
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
|
|
FRANK_ARCHON_PREFILL_INTERLEAVE=0
|
|
# Prefix cache: pinned host memory only. The disk tier was removed outright
|
|
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
|
|
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
|
|
# entry tracks the cache dtype (58.50 KiB at fp8). Watch prefix_hit_rate after
|
|
# this change: a drop is extra prefill, and prefill is exactly the traffic that
|
|
# costs decode its cadence.
|
|
FRANK_ARCHON_PREFIX_CACHE_MIB=8192
|
|
# Frank's pin law: the PLE n-gram table serves from pinned host RAM the cards
|
|
# read. `ram` REFUSES the start if memlock is too small — the unit's
|
|
# LimitMEMLOCK=infinity is the other half of this law.
|
|
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
|
# TTM preflight. The pinned ledger rides TTM's shared pages, and the kernel
|
|
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
|
|
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
|
|
# itself before refusing by name, so recovery is a boot, not a visit.
|
|
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
|
|
# per the bats ledger test; at the 45,568 pool that computes to 150,200 MiB, so
|
|
# the floor sits just above it.
|
|
FRANK_ARCHON_TTM_MIN_MIB=152000
|
|
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
|
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
|
|
# this unquoted on purpose, so keep multi-flag values as one string.
|
|
FRANK_ARCHON_EXTRA_ARGS=
|
|
# Deployer gate: a truncated fetch must be a fast refusal, not a long-timeout
|
|
# boot. Floor sits under the live payload and under every rollback payload
|
|
# (see the page's Eras table for what is on disk and what needs a re-download).
|
|
FRANK_ARCHON_RAD_MIN_MIB=110000
|