kv_attn 26.00 KiB fp8 pages at 607,740 tokens (identical to the w4dn8
boot), slab 17,610/17,658 slots per rank (above w4dn8's 17,548 — ten
bf16-protected experts per layer pack more of the 4-bit table), sampler
9 rows, MTP 2/step, KV lending 4.22 GiB, prefix tier recovered 900,176
blocks across the payload swap. Vision probe green (reasoning read the
solid-red PNG; empty content at 64 tokens = the thinking-budget law).
e2e gate RED on three dankclaw specs: Marreta was down since the 01:41
dankclaw reboot (marreta-browser dependency-failed at boot beside
gluetun, not re-armed); pair restarted manually, redeploy rerunning
the gate.
New runbook frank-archon-radiance-qwen38-iq4r.md (publisher container,
payload chain, shape-verbatim cutover record with the boot-ledger and
measurement slots left open for the first boot, rollback). LLM index,
root vault index, and AGENTS.md (fleet table row, section, runbook
index: iq4r ACTIVE, w4dn8 RETIRED/on-disk, fp8-native deleted). Full
suite green.
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape:
prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k —
the third slab step bought nothing, so the slab is a threshold whose cliff is
the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent
streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in
decode concurrency, the axis the shape hunt never measured because probe leg
B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode,
512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 ->
3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %,
wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a
threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement
it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are
not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s.
Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at
29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable).
The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the
number, so the next resize starts from the engine's arithmetic.
Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool
hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism,
the pinned pool is the slow path. Capacity contract live-verified at 3 slots:
3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too
large to ever fit.
Probe law learned: the oversized-prompt refusal names the prompt's token count
BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on
prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.
Probe on the live service: four concurrent 196,960-token chats, 4/4 exact
marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the
single-stream ceiling, so the four streams shared one pipeline and the KV pool
never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008.
Recorded with the caveat that 8 generated tokens do not make a decode rate.
Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.
The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).
Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
- AGENTS.md Model Config: the Protoss names are now the ids on the wire
(per-endpoint knob named); verify a rename on /v1/models, never on a chat
response — radiance echoes the requested name
- OMP Harness.md: harness id == served id; one id per model everywhere
- Homepage.md: the docs-page "api id" annotations and the switchover promise
are retired
- OpenWebUI.md: the id-keyed trio (model row, access_grant, api_configs
model_ids) is a foreign-key set — re-key it in the same deploy; plus the
contenteditable-composer probe note
- frank-secondary/frank-large/frank-primary runbooks: served ids
frank-zealot/frank-dragoon/frank-archon, and the bench.conf drop-in that
made every frank-primary deploy since 2026-10-08 04:09 restart the staging
worktree behind a green health gate
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
Deploy verified: nvidia-smi 21,436 MiB used / 2,689 free at ubatch 2048 -
the MiMo-priced projection (~21,430/~2,700) landed within 11 MiB, and the
verbosity-4 ledger confirms the KV law to the MiB (CUDA0 KV buffer
12,544.00 MiB = K(f16) 6,272 + V(f16) 6,272 = 32 KiB/token over the
401,408 pool), compute buffers 1,136 MiB CUDA0 + 848 MiB host, mmproj
879 + 248 MiB. Playwright gate: 33 passed.
Two lessons recorded: the 203/EXEC boot loop my shebang-less rewrite
caused (journal signature + the head -1 check), and that llama.cpp's
"n_ctx_seq < n_ctx_train" line is informational when the context is BELOW
the trained window - the Gemma4 drafter warning was the opposite case.
The unit's Description named the Swift-1.5 era (and said 2x131072/q8_0/
draft-mtp); it now names the Q6_K shape. Cosmetic, so it is NOT deployed
as its own restart: the next frank-secondary deploy relinks it.
Co-Authored-By: omp <omp@franksdank.cloud>
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.
Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).
Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.
Co-Authored-By: omp <omp@franksdank.cloud>
Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the
exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v
(6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at
n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the
byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads,
not 8.
Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is
funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus
variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0;
vision probe returns Red at prompt_tokens 151.
Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B)
+ the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/.
Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the
24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch
1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk,
not loaded: its draft KV is built against the target's full window,
measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise).
Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off =
21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319.
Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5
full-attention layers) so KV scales with 5 layers only, and
attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the
measured q8_0 pool = 2.05 GiB = the K-only figure).
Serve script gained VERBOSITY (+ --log-verbosity in the argv), the
SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4.
deploy-remote.sh payload gates follow the env. The frank-secondary bats
seam is rewritten for the new contract (argv-based) and fixes a
pre-existing vacuity: serve_argv regenerates the test env, so
mutations made before it were silently undone - tests that mutate now
exec through run_stage. frank-secondary is the fleet compaction hub
(200,704 is the only window that fits a worst-case compaction input).
Execution facts, blocker trail (plugin-tap probe law, weights-slab
formula, KLD_MOUNT), engine verdict, and the post-chain order. Linked
from the docs index per the completeness seam; the cutover runbook
replaces it.
23,120 used / 1,005 free at boot, 763 settled after a vision probe
(encoder buffers resident). Rung cost 746 MiB, not the transferable
616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode
unchanged as the law says. 2048 = ceiling for this shape.
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.
serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).
frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
AGENTS.md: row + section carry the measured 177,109 MiB container, floor
177000, live cutover 2026-10-06. Runbook: the three convert laws the live
runs proved (scale-name, file-shadow, symlink-through), the measured boot
(125.26 GiB page-in 38.1 s, PLE pin 47.68 GiB/43.9 s, ~2 min to health),
the MTP/vision/tool/reasoning contract results, and the SSD tier fix.
Stop-window A/B, same 80k filler prompt, 128-token essays, runtime
timings: HALOGEN_FA_TQ=256 (upstream issue #12's carve workaround) moves
decode ±0.3% (noise, draft acceptance identical) and costs 9.5% prefill
on this topology. The depth collapse sits deeper than the FA tile knob;
the default stands. The harness's slow/advisor roles moved to
frank-primary (fp8 KV 7.32 KiB/token = 8.7x less KV read at depth).
Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s ->
warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING
'smaller than ONE full-context entry' sizes against the native 262,144
window, not our 200,700 slot — documented as expected in the checklist.
Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k
vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches
decode) and the AGENTS.md unit paragraph regains the unit-facts
parenthetical the reflow dropped.
Runbook reshaped: 2 x 131072 no-spec identity, slot-2 arithmetic (drafter ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), ngram-mod skipped on the recurrent-rollback law, MTP references kept (44.8-52.1% acceptance, 58.9-66.9 t/s), rollback = one revert of 812d9eb. AGENTS fleet row + section, LLM/index.md intro + entry, root index entry all flipped to the 2026-10-06 shape. 360/360 suite green; e2e gate 33/33 green at deploy.
Boot ledger lines, 19,404 used / 4,721 MiB free (the q8_0 proof), decode arms (66.1 t/s @0.8, 66.9 @0, 58.9 @recipe 1.0 with acceptance 52.1%/52.2%/44.8%), vision Red probe, reasoning_effort xhigh/high contract, e2e 12-crawl transient + full-gate green rerun. Unit Description updated in-repo; the relinked unit picks it up at the next deploy/restart (no bounce for cosmetics). 360/360 suite green.
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
workflows/ (12 KiB).
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
The parallel session's stash pop left GGUF Compatibility, Multi-Device
Splits, Serving Sizing, PCIe Link and Atomics, RDNA4 Container Serving,
vLLM Porting triage and RDNA4 MoE Budget unindexed and carrying 9
unresolved wikilinks (LLM Bench Windows / Model Swap / AGENTS targets
that never existed, a [[nodiscard]] C++ attribute parsed as a wiki link,
and two multi-line-wrapped link tokens). All repointed to real pages;
six rows added to the vault index; seam green.
The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
at /tokenizer (model-card docker-run parity; the engine's
HALOGEN_TOKENIZER default is the directory) — there is no
frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
fetch script size-checks them and sha-verifies only the two LFS
objects (checkpoint, tokenizer.json).
Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
Both indexes stated the pre-doubling VRAM as current. Now 2,279 MiB free with
the 4096/2048 pair named, so no index contradicts the runbook it links to.
Frank
