Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
a9da65e put the raise inside the serve script, which runs as `User=dank`: the
knob is root-owned 0644, so `-w` is false and the write silently failed — the
gate printed its refusal on a cap it had just failed to move. The privileged
path is systemd's `+` prefix: `ExecStartPre=+scripts/ttm-preflight.sh --raise`
runs as root even under `User=`, before ExecStart, so `Restart=always` can
never outrun it. The lib is now also executable (`--raise` sources the env file
and verifies), and keeps a `sudo -n` fallback for a human running the gate from
a shell.
Seam: the refusal cases run with a PATH that has `cat`+`bash` and NO `sudo`
(root writes a 444 file, so a passwordless dev box would have tested nothing),
and both now assert the gate's own exit 1, not merely non-zero. New seam covers
the unit line and the `--raise` entry point end to end (writable knob -> raised,
no args -> usage exit 2).
The TTM_TARGET_MIB guard I added on 1a6a78e was written `:"...` (no space), so
bash parsed the expansion as one word and executed `cmd :204800` as the
command `:204800`: exit 127 `command not found` on every start, before the
ttm gate ran. Caught on the deploy: the unit crash-looped 34 times in ~4 min
with exactly that journal line.
probe-frank-archon-kv.py now reads the slot count, window and KV dtype from the
deployed env on the host, so probe B exercises the capacity contract that is
actually live (4 x 200,000 today) instead of a hardcoded pair, and each probe's
marker starts with a unique word because show() scores the hit on that word.
The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit
from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of
RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve
script's preflight named the refusal and exited 1; Restart=always re-ran it
6,919 times over ~20 hours and frank-archon served nobody.
scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob
itself before refusing: the unit is root, the knob is a 0644 root module
parameter, and the cap is a runtime knob whose default is half of RAM — so a
manual sysfs write is not a repair procedure, it is a countdown to the next
reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate
re-reads the cap after writing and only then refuses, so a knob the kernel
will not move is still a named failure.
Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8
(quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds
the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut
the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so
the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the
measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are
untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the
25-row bound where the wht6 wire is bit-exact.
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.
The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).
Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
The KV pool at the fp8 shape backed 386,904 tokens — 1.93 sessions of the
served 200,000 window, so a second full-height conversation preempted to the
host/disk prefix tiers. Moving the cache to bf16 (the engine's only
unquantized rung; `--kv-cache-dtype` takes bf16|fp8 and has no per-tensor
K/V split) doubles what a token costs: 7.3125 -> 14.625 KiB per rank
(kv_attn 26.00 -> 52.00 KiB per 4-token page + kv_qsa_bkey 3.25 -> 6.50).
Pool 3543 -> 6912 MiB (437k tokens, >= 2 x 200,000), and the weights slab
pays for all of it: 26071 -> 22208 MiB, the pair pinned to the measured
card ceiling 29,120 MiB (card 31,856 - 1,710 consumed outside both budgets
- 1,024 free). The resident set that leaves VRAM spills into the pinned host
pool, so 40960 -> 44928 MiB, and the prefix host tier doubles with the entry
size (58.50 -> 117 KiB): 4096 -> 8192 MiB. Pinned ledger ~147 GiB, ttm floor
140000 -> 150000 (cap 200 GiB).
--max-num-seqs 3 -> 2: two slots x the window is the capacity contract, and
the "seqs 2 cannot boot" law is the retired tcclaviger fork's torch.compile
query_start_loc constraint, not radiance's (radiance booted seqs 2 in the
fp8-native era; the 2026-10-06 sweep had 2 and 3 tied inside +-3%). Bonus:
the fused all-reduce is now declared over 2 x 3 = 6 rows, under the 25-row
bound at which the wht6 wire is bit-exact.
Cost, paid twice on this engine: idle KV VRAM is also the expert cache's
spillover (lends 2.69 GiB + 740 MiB of arena), so the pool steals weight
residency from the slab AND the lendable cache. Priors: 2.14 GiB of stolen
weights cost 2.5% decode (MAX_MODEL_LEN 262144), the EVC ladder's other
direction cost 8.9%. Prediction -3..-8% vs the fp8 control (136 t/s at 44.5k,
acceptance 0.7246); the deploy boot ledger and live probes are the referee.
Seams: MAX_SEQS/HOST_POOL/KV_DTYPE/PREFIX_HOST literals follow the shape, and
two new invariants guard it — the KV pool must back seqs x max-model-len at
the declared dtype (an engine hard limit: an over-long request is accepted
then refused as too large to ever fit; the formula reproduces the fp8 boot's
386,904 to within 0.1%), and the VRAM pair must stay inside the ceiling.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
- AGENTS.md Model Config: the Protoss names are now the ids on the wire
(per-endpoint knob named); verify a rename on /v1/models, never on a chat
response — radiance echoes the requested name
- OMP Harness.md: harness id == served id; one id per model everywhere
- Homepage.md: the docs-page "api id" annotations and the switchover promise
are retired
- OpenWebUI.md: the id-keyed trio (model row, access_grant, api_configs
model_ids) is a foreign-key set — re-key it in the same deploy; plus the
contenteditable-composer probe note
- frank-secondary/frank-large/frank-primary runbooks: served ids
frank-zealot/frank-dragoon/frank-archon, and the bench.conf drop-in that
made every frank-primary deploy since 2026-10-08 04:09 restart the staging
worktree behind a green health gate
The 2026-10-08 rename moved the harness model ids and the homepage cards;
the served aliases on the wire stayed frank-primary/frank-large/
frank-secondary. This closes the gap: the alias each endpoint announces is
now the family name, while the services, units, scripts, env files and
runbooks keep the old names (container identity stays stable).
- scripts/frank-secondary.env ALIAS, scripts/frank-large.env
HALOGEN_MODEL_ID, scripts/frank-primary.env FRANK_PRIMARY_SERVED_NAME
(+ the two serve-script defaults/gate messages)
- seams re-pinned: frank_secondary/frank_large/frank_primary unit bats
- fleet harnesses retargeted: bench-probe, deep-probe, calib-push,
fleet-probes-swift15, autoresearch(+measure), bench-threads-batch-http,
bench-decode
- shared/homepage/docs.html: api-id annotations and the "switchover will
rename the ids" footnote removed — the page's names are now the ids;
opencode/Pi/curl examples send frank-zealot
- dankclaw/hermes/config.yaml.example: frank-large -> frank-dragoon and the
live 262144 context (Hermes routes by served id; see Hermes.md)
Consumers with live state re-keyed out-of-band in the same window:
OpenWebUI (model row, access_grant, openai.api_configs model_ids) and the
Hermes config.yaml.
The panel title carries a right-aligned 'AI guide ->' link to
/docs#models (the running-models section), at the point of use.
Bare mode recolors the link; layout is flex baseline, doctrine-minimal.
Seam: homepage_config.bats 6/6; headless smoke census + screenshot.
- Icons: Copyparty -> improvised amber folder, Marreta -> crumbling
wall (bricks + red crack), Calibre-Web -> book, OpenWebUI -> official
logo.png from openwebui.com, Seerr -> official seerr.dev
os_logo_filled.svg.
- Tiles with an open link are clickable whole (.panel.clickable: hover
lift + glow; inner link clicks pass through).
- Any DOWN panel (app tile or model card) gets a red gradient wash and a
slow pulsing glow (.panel.down, .mi-card:has(.badge.down)).
- docs / admin / account-settings buttons moved from the footer to the
header (nav.top-links); footer keeps the deployment stamp.
- Zealot compaction advice promoted from the Use paragraph to a bordered
glowing model.compaction callout (.mi-comp); component seam pinned to
the new field.
- Every app card carries hex-clipped art: 19 amber-tinted brand marks
from the Simple Icons CDN plus 19 improvised monogram SVGs, wired via
a per-service "art" field; homepage_config.bats pins art presence for
all 32 apps.
- Model cards render zealot -> dragoon -> archon (registry order pinned
in the seam).
- Dragoon and zealot share one common uncensored legal warning (adult
content, harmful/hateful/illegal descriptions, sensitive offensive
cybersecurity topics, severe legal risk, "don't ruin this server for
everybody"), hover label on card and docs.
- Archon gets a bordered red permission callout: ask Frank before use.
- SearXNG obfuscation-not-anonymity note moved to a card-level hover
warning (.svc-warn) and a docs hover note.
- Gitea: self-service signup flow; no need to ask Frank.
- Footer docs/admin/account-settings become glowing bordered buttons;
the admin button is shown only to homepage-admins members via a new
Caddy /whoami endpoint echoing the outpost-injected X-Authentik-*
headers (spoof-proof; the /admin* 403 stays the real gate). e2e pins
both sides.
- Docs naming swept: Protoss names primary, bracketed api ids, explicit
note that a future switchover renames the ids themselves.
The Protoss model cards render as .mi-name inside one full-width panel;
the panel-name census now counts both render styles, keeping services.json
the single source of truth.
The model cards become the StarCraft trio: frank-large -> frank-dragoon,
frank-primary -> frank-archon, frank-secondary -> frank-zealot (family
names only; endpoint/API ids unchanged). The three cards fuse into one
Model Information component: per-unit Protoss accent colours and hex-
clipped unit art, 15px type, clickable StarCraft unit-voice quotations
(original .ogg files from the fandom wiki, served from /sfx/), supervision
notes (dragoon/archon safe for autonomous use, zealot tight supervision),
compaction-model recommendation for frank-zealot, and uncensored-model
legal notes rendered as hover tooltips instead of wall text. docs.html
carries the same renames plus api-id mapping.
App-card updates: Seerr, Calibre-Web, RomM, Jellyfin set to WIP: DOWN
with Open links removed (new wip "down" badge state); Copyparty notes it
serves all other media apps content including WIP apps; Gitea and Plex
noted as needing logins separate from SSO; SearXNG noted as obfuscated,
not anonymous (upstream API keys tie to Frank).
Plumbing: Caddy @data allowlist grew /sfx/*; deploy_homepage copies sfx/;
homepage_config.bats pins the new probe names, the component fields,
art/voice-file presence, and the wip-down schema.
Every fleet model card (and the #models docs section) now states, in
plain family-facing terms, what the model is equivalent to:
frank-large ~ Claude Sonnet 5 / OpenAI-Luna, frank-primary ~ Claude
Opus 4.8 / OpenAI-Terra (both Summer 2026 frontier), frank-secondary ~
last year (2025) frontier. Rendered as a new details.equiv line
(desc-equiv, styled in the head color, bare-mode clean); pinned in
homepage_config.bats.
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
Frank