The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.
1862dc3 repriced the Ornith rung (nine full-attn layers: f16/f16 = dead
boot) and flipped CACHE_TYPE_V to q8_0, but its note that the seam was
'retargeted in 8f53df7' covered only the payload paths — the KV test
still asserted the 2026-10-03 unquantized-both-halves contract and held
the suite red. Test now pins the shipped contract: K=f16, V=q8_0,
flash-attn on.
The default-project-set rewrite dropped non---project args (e.g. --list),
so 'run.sh --list' silently RAN the whole suite instead of listing.
Args now ride alongside the default set; an explicit --project still
takes over. Verified: default --list = 33 tests listed; opt-in --list =
34 with the frankbook walk.
The rogue session's swap left this env reprice uncommitted in the tree
(2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn
layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16
lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because
the compaction invariant needs slot >= 191,148. The only lever that
keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K
stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch
script manifest already flipped in 3ee88d7. MEASURE after boot; under
1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
The 17:47 Ornith-1.5-9B swap (77decb5) updated frank-small.env and
serve-frank-small.sh but left two consumers on the MiMo-Q6K paths:
deploy-remote.sh's LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ defaults (the
payload-existence gate checked files the unit no longer loads) and
frank_small_unit.bats (stubs + env assertion), which held the whole
bats suite red. Both now point at the Ornith-Q8 payload; docs updated
with the conditional frankbook e2e project.
The rogue session's Ornith commits (3ee88d7/77decb5) swapped the env and
fetch script but left the deploy handler gates and the seam asserting
the MiMo-Q6K paths — the seam failures 195-203 and a deploy that gates
a file the unit no longer reads. Flip both to Ornith-Q8 (Q8_0 GGUF +
BF16 mmproj), keeping the anti-drift twin structure intact.
The 11-admin spec walked links that terminate on FrankBook-Pro
(192.168.0.112), which sleeps when idle — every deploy gate timed out
there (measured 2026-10-05, two deploy windows). Split: the frankbook
links move to tests/e2e/specs/19-frankbook.spec.ts under a frankbook
project that run.sh excludes by default and deploy.sh triggers ONLY
when a frankbook blast-radius path changed (the spec, frankbook-pro/,
the homepage registry, the Caddyfile arr vhosts). deploy.sh computes
the trigger from the same BASE..HEAD revision range the plan uses.
The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
at /tokenizer (model-card docker-run parity; the engine's
HALOGEN_TOKENIZER default is the directory) — there is no
frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
fetch script size-checks them and sha-verifies only the two LFS
objects (checkpoint, tokenizer.json).
Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
curl drops the Range header across the HF /resolve/ 302, so '-C -'
against the resolve URL pulls the 1 KB redirect body and exits 0 with
the file still 14 MiB short (measured 2026-10-05). Resolve the
effective URL first; the Xet CDN honors Range.
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest
dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the
.hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2)
on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law).
Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images)
and the censored base (rollback = revert this commit; payload, fork
build and build-rocmfpx.sh pin stay on disk).
- serve script: podman run under the root unit, exec podman wait so
Restart=always is real; gates image digest, checkpoint byte-count,
flat regular-file tokenizer, GPU nodes; engine stdout to log.
- deploy handler: gates the artifacts, sweeps the legacy bare
llama-server + old container name by NAME (never pkill llama-server:
frank-small), retires the 2026-09 quadlet, port cutover 7770.
- fetch script: -C - resume, -f, byte count, sha256, --verify-only.
- seam rewritten (17 tests): halogen is now the deployed runtime.
Both indexes stated the pre-doubling VRAM as current. Now 2,279 MiB free with
the 4096/2048 pair named, so no index contradicts the runbook it links to.
The AGENTS.md lesson "`--ubatch-size` has an OPTIMUM on this stack" was load-
bearing and over-general: it was measured on the 2x R9700 layer split, and the
same knob on the CUDA 3090 behaves the opposite way (curve climbs monotonically
to 2048). That difference is now the reason frank-small ships ubatch 2048, so
the lesson names the backend it was measured on and points at the counterpoint.
Adds the 2026-10-05 boot-pair A/B to the runbook (+4.6 % prefill at 8,928
tokens for 616 MiB, decode flat, the sub-ubatch prompt as the control), marks
the superseded 21,276/2,849 VRAM row, and logs the day. The old "ubatch 2048
failed the scratch boot" line stays visible as what it was: a predecessor-shape
observation, not a law.
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048.
Why the ubatch move is right on THIS card: the prefill curve is
card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512
and every step up costs GiB for no speed; on this CUDA 3090 it climbs
monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423
t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung.
The logical batch is free either way: two sweeps agree that at a fixed
ubatch, 2048/4096/8192 give identical prefill speed and identical peak
VRAM, because the compute buffers are sized by the ubatch alone.
The risk is all in the ubatch, and it is documented rather than assumed:
the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this
replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch
2048 failed the scratch boot" with MORE free than that. Compute buffers
are allocated at graph capture during boot, so a failure is loud and fast;
the step-down rungs (4096/1536, then 4096/1024) are named in the env.
The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and
UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test
asserts the argv follows the shipped env plus the llama.cpp invariant
ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps
the seam green, inverting the pair to 2048/4096 turns it red.
rad-convert --plan-only with the deployed orcarouter FP8 payload:
'no architecture plugin claims qwen4exp at quantisation fp8; served at
(unquantised)'. Our quantization is reproducible from a bf16/abliterated
source by recipe (libr4d reads fp8 experts), not from our file. Image
pulled on dankserver, no GPU touched.
The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.
Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.
Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
Deploy b820d34, measured: --no-enable-prefix-caching in the engine argv,
health 200, warm boot 76.57 s (compile cache intact), expert plan
unchanged (17.77 GiB / 7,762 slots), GPU KV cache 424,678 tokens (up
from 401,408 at the same byte ask - pool now prices ~7,456 B/token with
the per-block mamba checkpoint pages gone; still clears the 400,700
floor), zero gdn_hip lines, and the 40,771-token boundary probe
(13 x 3136 = 40,768 crossed) decoded the 1-400 count coherently
(0.997 monotone, tail ...398 399 400). e2e 32/33 - the failure is the
admin spec's FrankBook-Pro leg (mini asleep), environmental.
Two bash defects in the 9d8b185 prefix-cache wiring, both measured live
on the 2026-10-05 deploy attempt (unit crash-looped 190x, exit 127,
endpoint down for the whole 3600 s health window):
1. A # comment block inside the backslash-continued podman run
terminates the continuation: the comment swallows the joined line
and the flag line then runs as its own command
('--no-enable-prefix-caching: command not found'). The comment moves
above the run block, with a no-comments-in-chain note.
2. The ${VAR:+A}${VAR:--B} double-expansion is ambiguous: when the
variable is set, ${VAR:--B} parses as a SUBSTRING OFFSET, not the
:- default — measured render was '--no-enable-prefix-caching1'.
The flag is now branched explicitly into
FRANK_PRIMARY_PREFIX_CACHE_FLAG (exactly one of the two flags
reaches the engine; unset keeps the historical flag).
Seam: the render test sources the shipped env through the branch and
asserts the exact flag word the engine receives (render-assert, the
same pattern the spec knobs use). bash -n clean; all 22 frank-primary
seams pass.
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.
Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).
RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.
Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.
Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
The host region was the ONE arena-plan term still autofitting per boot: with
no cap the planner binds it to MemAvailable AT BOOT minus PLE/process/api/
margin terms (boot 2026-10-05: 237.97 GiB -> 159.39 GiB usable; the same host
reads 75.4 GiB available while running). Demand is 105.48 GiB
(52.74/rank x 2 ranks, 23,040 swap-target experts); a boot where co-tenant
RAM squeezed the bound near that silently pins next-hottest experts VRAM-only
(pinned_extra warning, smaller swap region, no flag change). 120 GiB =
measured demand + 14% headroom on 246.1 GiB total. Semantics verified in the
shipped image: cap binds as the bound, exceedance only warns.
The GPU-side arena is unchanged (deterministic arithmetic over the same
constants): verify after boot that host reads '--expert-offload-mem' and
slots stay 7,762. Seam: red first (wiring + value), then green; full suite
368+1 green.
Verified in the shipped image (v1/worker/gpu/cudagraph_mem.py):
estimate_cudagraph_memory() checks VLLM_CUDAGRAPH_MEM_ESTIMATE_MB FIRST and
returns that MiB total outright, ignoring the BASE/PER_GRAPH pair — the only
way to state the CUDA-graph plan term directly instead of deriving it. We
keep shipping the pair (0/12) so a future spec-decode shape re-prices
automatically; documented the switch condition.
Also corrected two stale shape-G claims in the runbook that predated the
1.00 revert: the deployed shape block and the lever table both still said
util 1.0 / 'le=1 ceiling', while shape G ships 0.982.
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used /
2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on
Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card),
512px red-PNG vision verified, and the new probe-size law (>=256 px: a
10x10 image undersupplies image tokens and the model answers 'video'),
nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides)
- frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and
becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured)
- AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index
(active row + two retired rows); LLM + root index entries
- Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical
to Q5_K_M - the vision probe scare was probe size, not payload
The freed third slot funded exact V; the remaining ~3.5 GiB of honest
nvidia-smi free (3,571, not the 4,046 total-minus-used I quoted - the
driver holds ~450 MiB of reserve) funds the +875 MiB of Q6_K weights.
Projected landing ~2.7 GiB free, inside the 2-3 GiB floor; measure on
boot, step the pool one 12,288-token increment if it lands under 2 GiB.
- payload dir MiMo-Q6K/ made self-contained (mmproj hardlinked from
MiMo-Q5KM/, link-count-2 verified)
- env/serve/deployer/bats repointed; download script now fetches and
verifies Q6_K (7,793,710,624 B, ef96d05a...) with Q5_K_M documented as
the rollback; payload sha verified live on the host before deploy
- seam green 14/14 against the stub engine
- new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the
Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors),
27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain
rejection (-15% prose: recurrent-state rollback disabled), the
bench-window hardening laws
- frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected
(gufo payloads removed from disk 2026-10-03 - that rollback is now a
re-download; the aligned Q8 file is the direct one, still on disk)
- frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully
unquantized f16 KV: boot-pair table gains the measured row (20,530 used /
4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to
the 3x180224 + q8_0-V shape documented
- AGENTS.md: fleet table rows, both endpoint sections, the chain law
extended with the recurrent-model measurement, the --parallel law with
the reslice, runbook index (active + retired)
- LLM/index.md + root index entries updated
Record the live numbers and the UTIL 1.0 verdict (unbootable: worker init
demands free > budget; ValueError on both ranks before a single weight loads).
Driver used 30.97 of 31.86 GiB; the 0.89 GiB/card margin held through a live
4,096-token generation at ~38-39 tok/s.
Worker init refuses budget >= device total on BOTH ranks:
ValueError: Free memory on device cuda:0 (31.29/31.86 GiB ...)
NRestarts=2 crash loop, zero weights loaded. Experts stay at the v1 shape
(graph term 0.07 GiB, KV 3.1665e9 = the 400,700-token ask); only the budget
returns under the law. Arena 17.78 GiB / ~7,912 slots predicted.
Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot
reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was
fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0,
the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target
18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at
the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver
high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is
UTIL=0.982.
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
Frank's order, two messages: "extend the max context of frank-small to
200700 per individual stream without further quantization", then
"reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to
1k", then "unquantize the KV if we have extra space".
CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704:
above the 200,700 the harness compacts at, so a full-length request
lands in one stream instead of a 400 exceed_context_size_error. The pool
must divide evenly across slots and stay 256-aligned for the FA path,
which is why it is 401,408 and not 401,400.
CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for
unquantized values (24.5 -> 32 KiB/token), and the budget lands back in
the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of
the card idle. The asymmetric-KV contract that created this endpoint is
spent with it; both --cache-type flags stay explicit.
--batch-size/--ubatch-size are now passed (2048/1024). The endpoint had
never passed either, so it ran on engine defaults nobody wrote down.
Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
Frank's order: replace frank-large with
kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF's
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf, following that card.
The file is ftype 114 — 339 GGML_TYPE_Q6_0_ROCMFPX (102) + 159
GGML_TYPE_Q8_0_ROCMFPX (103) + 353 F32 over 851 tensors,
qwen35/64 layers/full_attention_interval 4/4 KV heads/262,144 window —
read out of its own header. The pinned ROCmFPX build already knows both
types (ggml.h 102/103 plus convert/cpy/getrows/mmq/fattn device paths
for 102), so no rebuild: the fork stays at 36e9acd and the payload
directory is the only new host state.
Card compliance, all already our flags: Q4_0 draft head (not Q8_0),
--spec-draft-n-max 4 (the knee on both cards), --spec-draft-ngl 99,
--jinja, --fit off, -fa on, -dio, one slot at full window, thinking with
the card's own sampling triple (which the file also embeds as its
general.sampling defaults). Kept from the deployed shape: the BF16
projector (this repo ships only a Q8_0 one; BF16 is the file verified
4/4 on spatial ground truth here) and the byte-identical MTP head, which
the new script hardlinks instead of re-fetching 1.6 GiB.
Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md
Four harness gaps that block an honest MTP re-measurement:
- measure: AR_TEMPS sweeps sampling temperatures (default "0,0.7"). Draft
ACCEPTANCE is sampling-dependent — a greedy draft matches a greedy target
exactly and is rejected more under a non-greedy target — so judging a spec
arm only at temperature 0 judges half the contract. The unsuffixed keys stay
temperature 0, so every historical number remains comparable.
- window: capture an evidence.log sidecar (expert-plan lines, compile path,
graph sizes, /metrics spec-decode and cache counters, OOM signatures). The
arena/overhead/reserve numbers exist ONLY in the boot log; an arm that loses
decode speed is otherwise unexplainable afterwards.
- orchestrator: FRANK_PRIMARY_AR_ENV_FILE stages an env other than the repo's,
which is how arms A/B without mutating the checkout (the fingerprint covers
whatever is staged, so the memo can never answer one arm with another's
numbers), plus METRIC lines for the swept temperatures and the evidence dump.
- scripts/autoresearch/frank-primary-arms.sh: runs a sequence of full windows
(baseline, MTP3+probabilistic with a step-4 capture ladder at precise memory
ON, the same at the default overhead, and the vendor SEQS-16 shape), gating
each staged env on its JSON knobs still parsing AFTER sourcing — an unquoted
{"temperature":0.7} loses its quotes at source time and reaches the engine as
{temperature:0.7}, which is a 20-minute boot wasted.
Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.
The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.
Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
Live after the 1ac4d48 deploy: 'expert_precise_memory': True, runtime overhead
5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB /
5986 slots, GPU KV cache 398,658 tokens / 1.99x, pool 3,150,000,000, MNBT 1024
-- the probe's projection is now production.
Compile cache proven on consecutive boots of that shape: cold 'Compiling a
graph for compile range (1, 1024) takes 179.24 s' / 'torch.compile took
188.26 s'; warm 'Directly load the compiled graph(s) ... from the cache, took
2.143 s' / 'torch.compile took 3.97 s in total' / 'init engine ... took 41.13 s
(compilation: 3.98 s)'. Host dir 1.4 GB / 4,707 files. Time to /health stays
payload-bound (490 s warm), so the claim is ~184 s of inductor/AOT build per
restart, not a proportional boot cut.
Tests: 367/367.
Measured by boot probe on the shape-D base (production stopped, candidate
booted AS the service), plan numbers read off the engine's own log:
runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry)
expert cache 14.87 -> 17.22 GiB
slots 6,496 -> 7,522 (+1,026, +16%)
swap region 13.7 GiB / 5,986 slots
boot healthy in 420 s, no OOM
'--expert-precise-memory': True also appears in non-default args, so a silent
argparse drop is visible. The pool is NOT cut (option A) because the pool is
not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency,
and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded
as the conservative fallback.
Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no
/root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile
artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral
layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays
'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting
FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing.
GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B
measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes
needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB.
Tests: 367/367.
`deploy.sh <target>` with no `--service` also walks the per-service revision
markers and restarts everything behind HEAD whose paths changed — a drain, not
a sync. That is how the latent plex gate bug surfaced: the backlog fired all
at once and plex's http:// readiness probe, which can never succeed against a
PMS that drops plain HTTP off-loopback, burned its full 600 s and aborted the
run with every later service left pending.
Both facts recorded: AGENTS.md gains the drain law under Deployment, log.md
gains the plex entry with the evidence chain.
deploy_plex gated on `curl http://127.0.0.1:32400/identity`, which can never
succeed: Plex accepts the TCP connection (it arrives on the container's eth0)
and then answers with nothing, so curl reports "Empty reply from server" (52).
Measured 2026-10-02 on a plex that was healthy the whole time —
`https://127.0.0.1:32400/identity` returned 200 in 44 ms while the gate burned
its full 600 s and failed the deploy.
The repo already knew this law in another place: tests/caddyfile.bats pins the
plex vhosts to a TLS upstream with the comment "PMS drops plain HTTP
off-loopback". The deployer's gate never got the same treatment, and plex's
per-service revision had been behind long enough that the gate had not run
against this posture until now.
-k because PMS serves its own self-signed cert on 32400.
index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and
frank-small at 2 x 262144; the active-deployment list had no frank-primary
entry at all. Fixed both, plus a malformed GLM list item.
log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever
measurement, the 2x2 probe table, the measured 1.99x), and the frank-small
entry's stale consequence bullet now points at the resolution.
frank-small runbook: the compaction-fit note is corrected - the 200704 window
moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
Comments/docs only - no functional change. The live boot of c769ff2 reported:
GPU KV cache size: 398,658 tokens,
Maximum concurrency for 200,704 tokens per request: 1.99x
expert cache space 14.87 GiB / 6,496 slots of 2.344 MiB
The arena prediction (6,496) landed exactly; the concurrency prediction (2.04x)
did not, because the per-token KV price is SHAPE-DEPENDENT: 7,701 B/token at
MAXLEN 262144 (6.1e9 -> 792,069 tokens) but 7,892 at 200704 (3.146e9 ->
398,658). Never extrapolate a concurrency figure across geometries.
Also smoke-verified: three concurrent 67-token requests answered in parallel,
~4.1 s each - the 3 slots are real, and the pool admits 2 full-length.
Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.
Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.
Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.
Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.
The 2 x 200704 fp8 / MNBT 1024 shape (30904ba) crash-loops at boot: the
engine dies in determine_available_memory -> profile_run -> _dummy_run with
the SAME torch.compile dynamo constraint that killed the bf16 2 x 262144
shape on 2026-10-01:
RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'
Not an OOM. Restoring the last shape measured serving (fp8 3 x 262144 /
KV 6.1e9 / MNBT 2048 / arena 6,248) so the endpoint serves again; the
desired edits are re-applied one at a time from this base.
Frank