rank 5. The three LLM handlers duplicated the same tail (relink,
daemon-reload, enable, restart, /health gate, MainPID port proof) ->
start_root_unit SERVICE PORT LIMIT; frank-archon's bespoke 900-iteration
loop was exactly wait_for_health at limit 900. copy_config DEST SRC...
replaces five hand-rolled mkdir+cp pairs (loki, alloy, grafana x2);
retire_unit_file PATH TRASH [UNIT] replaces the duplicated compose-era
unit retirements (halogen dragoon quadlet, comfyui.service). The health
grep seam now pins the per-endpoint call (start_root_unit frank-dragoon
7770).
rank 3. shared/services.yml is the single source of truth for every
deployable service (hosts, unit dirs, units, path claims, tests, scrape/
dns/probe facts, aliases, stacks); scripts/service-map.py is its only
reader (hosts/services/allows/plan/seams/owner/units/stack/get/render/
check). deploy-plan.py becomes a thin CLI wrapper; both --service
allow-lists (deploy.sh, deploy-remote.sh) and the changed-test mapper are
the inventory. Selection semantics: host scoping (a host's tree selects
only its own services), the quadlet netns/network coupling graph resolved
across directories (gluetun's riders live in their own dirs), stacks, and
canonical restart order.
Against the pinned old-planner fixture: 921 raw deltas reduced to 17, all
intentional (anubis now maps; cross-host paths select nothing; the retired
qwen-sharp template plans nothing and the live froggeric template plans
frank-archon; dankserver/romm/romm.container, missed by the old filename
rule, plans the romm trio). The fixture is regenerated as the new pin.
Integrity gates: schema, unit claims, quadlet-file ownership, path-glob
liveness, Pi-hole hosts / Prometheus jobs / DNS / homepage probes drift.
tests/service_inventory.bats pins all of it plus the plan fixture and the
deploy-plan CLI. Allow-list grep seams in hermes_webui/observability_quadlet
become behavioral allows calls.
frank-zealot shipped Restart=on-failure while its two siblings shipped
always. exit 0 is a CLEAN stop to systemd: on-failure leaves the endpoint
dark and the unit GREEN until the next boot — and frank-zealot is the
compaction hub for all three endpoints, so its quiet death degrades every
conversation in the fleet.
tests/llm_units.bats gains the fleet parity seam (Restart=always,
RestartSec 1..20, WantedBy=multi-user.target across scripts/frank-*.service);
the per-unit seams drop their weaker Restart greps so the policy lives in
one place. Runbook unit line updated.
frank-dragoon's health gate never checked WHO answered :7770 — the
false-green the fleet learned about on 2026-09-26 (a stray probe holding
the port satisfies a bare curl while the unit is stopped). The other two
handlers carried that check as inlined copies; all three now call
assert_unit_owns_port, and dragoon gets the protection it was missing.
The seam follows: the endpoints' text-grep deployer tests are deleted,
because tests/llm_deploy_gates.bats EXECUTES the same handlers end to end
(stub systemctl/podman/sudo/ss/curl) and asserts refusals, restarts, unit
links, the pending marker, and the MainPID hardening. GATE_PREFIX is
exported at each handler for the same reason the serve scripts export it.
deploy_llm_dragoon/zealot/arkon now call require_image / require_bytes /
require_min_mib / require_file / require_dir_files, so the deployer and the
serve scripts refuse the same missing payload with the same words — the
2026-10-01 crash-loop law, enforced by construction instead of by duplicated
shell. require_exec joins the library for the native-runtime gate (a
compiled tree that lost its +x refuses like a missing one). relink_unit
takes SYSTEMD_SYSTEM_DIR so a handler can be driven end to end in a sandbox.
The library's own refusals now carry their status: a bash function that
ends inside an if returns the CONDITION's status, so a gate whose last
statement is its refusal printed the message and returned SUCCESS — the
new seam caught it, and the gate tests pin it.
New seam tests/llm_deploy_gates.bash + tests/stubs/: deploy-remote.sh's LLM
handlers EXECUTED against stub systemctl/podman/sudo/ss/curl, asserting
exit status, refusal text, no restart, and the pending state marker.
The image/.rad/template gates are require_image/require_min_mib/require_file,
and the gates an operator can only reach on the host (GPU nodes, the ttm pages
cap, the run line) are now drivable: FRANK_ARCHON_DEV_KFD, FRANK_ARCHON_DEV_DRI,
FRANK_ARCHON_TTM_SYSFS and FRANK_ARCHON_LOG default to the production paths.
Seven new seam tests EXECUTE the serve script against fixtures (stub podman,
stub nodes, stub sysfs, temp payload) and assert exit status, the refusal
words and the engine argv. The old text-grep gate test is deleted: it pinned
the words of the source, not the behaviour.
The three -e gates (runtime, GGUF, mmproj) are require_file calls: the refusal
now names the artifact class, the exact path and the script that produces it.
The seam asserts that observable triple instead of the old sentence order.
The hand-written image/ck/tokenizer gates are require_image, require_bytes and
require_dir_files from scripts/lib/llm-gates.sh. The serve seam is unchanged
and still green (its podman stub now models `podman image exists`, the
primitive the gate uses). Behaviour preserved: every refusal names the
artifact, the endpoint and the fetch script.
The serve scripts and the deployer each hand-wrote the same missing/truncated
artifact gates; the 2026-10-01 crash-loop was those twins disagreeing
mid-cutover. scripts/lib/llm-gates.sh is the single place the refusal lives:
require_file / require_bytes / require_min_mib / require_dir_files /
require_image, each returning 1 and naming the endpoint, the artifact and the
remedy. New seam tests/llm_gates.bats pins the contract at the arguments-in /
status-and-stderr-out boundary.
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.
The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).
Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
The KV pool at the fp8 shape backed 386,904 tokens — 1.93 sessions of the
served 200,000 window, so a second full-height conversation preempted to the
host/disk prefix tiers. Moving the cache to bf16 (the engine's only
unquantized rung; `--kv-cache-dtype` takes bf16|fp8 and has no per-tensor
K/V split) doubles what a token costs: 7.3125 -> 14.625 KiB per rank
(kv_attn 26.00 -> 52.00 KiB per 4-token page + kv_qsa_bkey 3.25 -> 6.50).
Pool 3543 -> 6912 MiB (437k tokens, >= 2 x 200,000), and the weights slab
pays for all of it: 26071 -> 22208 MiB, the pair pinned to the measured
card ceiling 29,120 MiB (card 31,856 - 1,710 consumed outside both budgets
- 1,024 free). The resident set that leaves VRAM spills into the pinned host
pool, so 40960 -> 44928 MiB, and the prefix host tier doubles with the entry
size (58.50 -> 117 KiB): 4096 -> 8192 MiB. Pinned ledger ~147 GiB, ttm floor
140000 -> 150000 (cap 200 GiB).
--max-num-seqs 3 -> 2: two slots x the window is the capacity contract, and
the "seqs 2 cannot boot" law is the retired tcclaviger fork's torch.compile
query_start_loc constraint, not radiance's (radiance booted seqs 2 in the
fp8-native era; the 2026-10-06 sweep had 2 and 3 tied inside +-3%). Bonus:
the fused all-reduce is now declared over 2 x 3 = 6 rows, under the 25-row
bound at which the wht6 wire is bit-exact.
Cost, paid twice on this engine: idle KV VRAM is also the expert cache's
spillover (lends 2.69 GiB + 740 MiB of arena), so the pool steals weight
residency from the slab AND the lendable cache. Priors: 2.14 GiB of stolen
weights cost 2.5% decode (MAX_MODEL_LEN 262144), the EVC ladder's other
direction cost 8.9%. Prediction -3..-8% vs the fp8 control (136 t/s at 44.5k,
acceptance 0.7246); the deploy boot ledger and live probes are the referee.
Seams: MAX_SEQS/HOST_POOL/KV_DTYPE/PREFIX_HOST literals follow the shape, and
two new invariants guard it — the KV pool must back seqs x max-model-len at
the declared dtype (an engine hard limit: an over-long request is accepted
then refused as too large to ever fit; the formula reproduces the fp8 boot's
386,904 to within 0.1%), and the VRAM pair must stay inside the ceiling.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
The 2026-10-08 rename moved the harness model ids and the homepage cards;
the served aliases on the wire stayed frank-primary/frank-large/
frank-secondary. This closes the gap: the alias each endpoint announces is
now the family name, while the services, units, scripts, env files and
runbooks keep the old names (container identity stays stable).
- scripts/frank-secondary.env ALIAS, scripts/frank-large.env
HALOGEN_MODEL_ID, scripts/frank-primary.env FRANK_PRIMARY_SERVED_NAME
(+ the two serve-script defaults/gate messages)
- seams re-pinned: frank_secondary/frank_large/frank_primary unit bats
- fleet harnesses retargeted: bench-probe, deep-probe, calib-push,
fleet-probes-swift15, autoresearch(+measure), bench-threads-batch-http,
bench-decode
- shared/homepage/docs.html: api-id annotations and the "switchover will
rename the ids" footnote removed — the page's names are now the ids;
opencode/Pi/curl examples send frank-zealot
- dankclaw/hermes/config.yaml.example: frank-large -> frank-dragoon and the
live 262144 context (Hermes routes by served id; see Hermes.md)
Consumers with live state re-keyed out-of-band in the same window:
OpenWebUI (model row, access_grant, openai.api_configs model_ids) and the
Hermes config.yaml.
- Icons: Copyparty -> improvised amber folder, Marreta -> crumbling
wall (bricks + red crack), Calibre-Web -> book, OpenWebUI -> official
logo.png from openwebui.com, Seerr -> official seerr.dev
os_logo_filled.svg.
- Tiles with an open link are clickable whole (.panel.clickable: hover
lift + glow; inner link clicks pass through).
- Any DOWN panel (app tile or model card) gets a red gradient wash and a
slow pulsing glow (.panel.down, .mi-card:has(.badge.down)).
- docs / admin / account-settings buttons moved from the footer to the
header (nav.top-links); footer keeps the deployment stamp.
- Zealot compaction advice promoted from the Use paragraph to a bordered
glowing model.compaction callout (.mi-comp); component seam pinned to
the new field.
- Every app card carries hex-clipped art: 19 amber-tinted brand marks
from the Simple Icons CDN plus 19 improvised monogram SVGs, wired via
a per-service "art" field; homepage_config.bats pins art presence for
all 32 apps.
- Model cards render zealot -> dragoon -> archon (registry order pinned
in the seam).
- Dragoon and zealot share one common uncensored legal warning (adult
content, harmful/hateful/illegal descriptions, sensitive offensive
cybersecurity topics, severe legal risk, "don't ruin this server for
everybody"), hover label on card and docs.
- Archon gets a bordered red permission callout: ask Frank before use.
- SearXNG obfuscation-not-anonymity note moved to a card-level hover
warning (.svc-warn) and a docs hover note.
- Gitea: self-service signup flow; no need to ask Frank.
- Footer docs/admin/account-settings become glowing bordered buttons;
the admin button is shown only to homepage-admins members via a new
Caddy /whoami endpoint echoing the outpost-injected X-Authentik-*
headers (spoof-proof; the /admin* 403 stays the real gate). e2e pins
both sides.
- Docs naming swept: Protoss names primary, bracketed api ids, explicit
note that a future switchover renames the ids themselves.
The Protoss model cards render as .mi-name inside one full-width panel;
the panel-name census now counts both render styles, keeping services.json
the single source of truth.
The model cards become the StarCraft trio: frank-large -> frank-dragoon,
frank-primary -> frank-archon, frank-secondary -> frank-zealot (family
names only; endpoint/API ids unchanged). The three cards fuse into one
Model Information component: per-unit Protoss accent colours and hex-
clipped unit art, 15px type, clickable StarCraft unit-voice quotations
(original .ogg files from the fandom wiki, served from /sfx/), supervision
notes (dragoon/archon safe for autonomous use, zealot tight supervision),
compaction-model recommendation for frank-zealot, and uncensored-model
legal notes rendered as hover tooltips instead of wall text. docs.html
carries the same renames plus api-id mapping.
App-card updates: Seerr, Calibre-Web, RomM, Jellyfin set to WIP: DOWN
with Open links removed (new wip "down" badge state); Copyparty notes it
serves all other media apps content including WIP apps; Gitea and Plex
noted as needing logins separate from SSO; SearXNG noted as obfuscated,
not anonymous (upstream API keys tie to Frank).
Plumbing: Caddy @data allowlist grew /sfx/*; deploy_homepage copies sfx/;
homepage_config.bats pins the new probe names, the component fields,
art/voice-file presence, and the wip-down schema.
Every fleet model card (and the #models docs section) now states, in
plain family-facing terms, what the model is equivalent to:
frank-large ~ Claude Sonnet 5 / OpenAI-Luna, frank-primary ~ Claude
Opus 4.8 / OpenAI-Terra (both Summer 2026 frontier), frank-secondary ~
last year (2025) frontier. Rendered as a new details.equiv line
(desc-equiv, styled in the head color, bare-mode clean); pinned in
homepage_config.bats.
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
The @data path allowlist on the post-auth listener did not include /img/*,
so docs screenshots fell through to the index.html catch-all and the
browser got 200 text/html for every PNG (docs images broken live
2026-10-08). Pinned by the homepage_caddy seam.
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.
Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).
Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.
Co-Authored-By: omp <omp@franksdank.cloud>
Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B)
+ the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/.
Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the
24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch
1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk,
not loaded: its draft KV is built against the target's full window,
measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise).
Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off =
21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319.
Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5
full-attention layers) so KV scales with 5 layers only, and
attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the
measured q8_0 pool = 2.05 GiB = the K-only figure).
Serve script gained VERBOSITY (+ --log-verbosity in the argv), the
SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4.
deploy-remote.sh payload gates follow the env. The frank-secondary bats
seam is rewritten for the new contract (argv-based) and fixes a
pre-existing vacuity: serve_argv regenerates the test env, so
mutations made before it were silently undone - tests that mutate now
exec through run_stage. frank-secondary is the fleet compaction hub
(200,704 is the only window that fits a worst-case compaction input).
Engine v1.2.0 (Codeberg e95d397, Docker Hub tag 1.2.0, index digest
665775ed...) removes --expert-vs-cache-ratio. The explicit
--vram-weights-mib 26071 / --vram-kv-mib 3543 pair transcribes the live
1.0.13 boot ledger at ratio 0.82 (weights 25.46 GiB, KV 3.46 GiB per
card); bracket arms scale KV linearly in (1-r): 0.90 -> 1968, 0.95 -> 984.
Serve script, autoresearch harness, and bats gates follow; the bats
shape asserts are realigned to the deployed seqs=3/spec=2 sweep result
(95923d3, df6b5bb) they failed to track.
Frank's order the same night. The 1024->2048 rung measured +4.6 %
prefill for 616 MiB on the 3090 (2026-10-05 boot pair); the 112k cut
left 1,751 MiB free, so it rides with ~1.1 GiB predicted margin.
Decode is ubatch-invariant; batch 4096 >= 2048 keeps the seam happy.
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
The 2026-10-06 cutover crash-looped frank-large 8 boots (03:09:39-03:11:42):
the engine's pin gate refused to start while host MemAvailable was ~74.7 GiB,
3.34 GiB short of the 78.11 GiB it needs (62.11 GiB checkpoint + 16 GiB floor).
It self-resolved in ~2 min when the dying 27B engine freed RAM, but left the
host with only ~1.8 GiB MemAvailable - a restart would crash-loop again.
serve-frank-large.sh now checks MemAvailable against the pin gate BEFORE
podman run and WAITS up to FRANK_LARGE_RAM_WAIT (default 600 s) for the RAM
to clear, so a transient dip boots once instead of crash-looping; if it never
clears it refuses loudly. The 16 GiB floor is the engine's (not
HALOGEN_HOST_RESERVE_GIB=20, a separate overall-budget knob).
frank_large_unit.bats: FRANK_LARGE_MEMINFO seam + a refuse test. Runbook +
log: the pin-gate law, the incident, and the fix.
Auto cache-sizing is OFF on this box: the boot ledger refused it at
36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent
request re-prefills its whole 79-81k context (84.9-179.8 s measured on
serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin
HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the
documented override; the per-store reserve re-check keeps the host safe.
Also: the serve script's 'podman run -d > LOG' captured only the
container ID — the runbook's boot-ledger grep read a one-line hex file
and the ledger existed nowhere on disk. Foreground 'podman run' makes
the ledger land in the log file as documented; exit-code and ExecStop
semantics are unchanged. Test seam updated for both.
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
workflows/ (12 KiB).
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
FreeToken has no /metrics endpoint — the job scraped 404s forever, a
structural never-green, not an outage (the Lucebox DS4.1 precedent). The
LlmEndpointDown alert regex self-adjusts (frank-.* matches existing series
only). Comments updated; seam now asserts frank-primary's absence. If
FreeToken grows /metrics, re-add the job.
FreeToken's /health returns 200 with {"status":"loading"} while the
engine loads — the FastAPI front end binds before the scheduler is up.
Measured 2026-10-06: the deploy gate went green after 6s while 131 shards
were still paging in (vLLM 503'd during load, which made the bare curl
honest there). The frank-primary gate now loops until status:ok; the
MainPID stray-port guard is unchanged.
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201
built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201
(scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated
--no-deps install per docs/install_amd.md).
Shape (first-boot conservative): ONE R9700 (second card idle;
--tensor-parallel-size exists for future 2-card tuning), KV pool
--num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token
harness worst case and one native 262,144 conversation), MoE LRU
--moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens
36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml
flips reasoningContentField on the harness side). Experts + PLE pin
~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity.
Container identity renamed to the unit name (Frank's call): frank-primary;
vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image
gate becomes podman image exists (a local build carries no RepoDigests;
the old digest grep refused exactly this shape). Rollback tag:
frank-primary-vllm-shapeG-20261003.
1862dc3 repriced the Ornith rung (nine full-attn layers: f16/f16 = dead
boot) and flipped CACHE_TYPE_V to q8_0, but its note that the seam was
'retargeted in 8f53df7' covered only the payload paths — the KV test
still asserted the 2026-10-03 unquantized-both-halves contract and held
the suite red. Test now pins the shipped contract: K=f16, V=q8_0,
flash-attn on.
The default-project-set rewrite dropped non---project args (e.g. --list),
so 'run.sh --list' silently RAN the whole suite instead of listing.
Args now ride alongside the default set; an explicit --project still
takes over. Verified: default --list = 33 tests listed; opt-in --list =
34 with the frankbook walk.
The rogue session's Ornith commits (3ee88d7/77decb5) swapped the env and
fetch script but left the deploy handler gates and the seam asserting
the MiMo-Q6K paths — the seam failures 195-203 and a deploy that gates
a file the unit no longer reads. Flip both to Ornith-Q8 (Q8_0 GGUF +
BF16 mmproj), keeping the anti-drift twin structure intact.
The 11-admin spec walked links that terminate on FrankBook-Pro
(192.168.0.112), which sleeps when idle — every deploy gate timed out
there (measured 2026-10-05, two deploy windows). Split: the frankbook
links move to tests/e2e/specs/19-frankbook.spec.ts under a frankbook
project that run.sh excludes by default and deploy.sh triggers ONLY
when a frankbook blast-radius path changed (the spec, frankbook-pro/,
the homepage registry, the Caddyfile arr vhosts). deploy.sh computes
the trigger from the same BASE..HEAD revision range the plan uses.
The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
at /tokenizer (model-card docker-run parity; the engine's
HALOGEN_TOKENIZER default is the directory) — there is no
frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
fetch script size-checks them and sha-verifies only the two LFS
objects (checkpoint, tokenizer.json).
Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest
dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the
.hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2)
on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law).
Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images)
and the censored base (rollback = revert this commit; payload, fork
build and build-rocmfpx.sh pin stay on disk).
- serve script: podman run under the root unit, exec podman wait so
Restart=always is real; gates image digest, checkpoint byte-count,
flat regular-file tokenizer, GPU nodes; engine stdout to log.
- deploy handler: gates the artifacts, sweeps the legacy bare
llama-server + old container name by NAME (never pkill llama-server:
frank-small), retires the 2026-09 quadlet, port cutover 7770.
- fetch script: -C - resume, -f, byte count, sha256, --verify-only.
- seam rewritten (17 tests): halogen is now the deployed runtime.
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048.
Why the ubatch move is right on THIS card: the prefill curve is
card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512
and every step up costs GiB for no speed; on this CUDA 3090 it climbs
monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423
t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung.
The logical batch is free either way: two sweeps agree that at a fixed
ubatch, 2048/4096/8192 give identical prefill speed and identical peak
VRAM, because the compute buffers are sized by the ubatch alone.
The risk is all in the ubatch, and it is documented rather than assumed:
the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this
replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch
2048 failed the scratch boot" with MORE free than that. Compute buffers
are allocated at graph capture during boot, so a failure is loud and fast;
the step-down rungs (4096/1536, then 4096/1024) are named in the env.
The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and
UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test
asserts the argv follows the shipped env plus the llama.cpp invariant
ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps
the seam green, inverting the pair to 2048/4096 turns it red.
Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.
Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.
Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
Two bash defects in the 9d8b185 prefix-cache wiring, both measured live
on the 2026-10-05 deploy attempt (unit crash-looped 190x, exit 127,
endpoint down for the whole 3600 s health window):
1. A # comment block inside the backslash-continued podman run
terminates the continuation: the comment swallows the joined line
and the flag line then runs as its own command
('--no-enable-prefix-caching: command not found'). The comment moves
above the run block, with a no-comments-in-chain note.
2. The ${VAR:+A}${VAR:--B} double-expansion is ambiguous: when the
variable is set, ${VAR:--B} parses as a SUBSTRING OFFSET, not the
:- default — measured render was '--no-enable-prefix-caching1'.
The flag is now branched explicitly into
FRANK_PRIMARY_PREFIX_CACHE_FLAG (exactly one of the two flags
reaches the engine; unset keeps the historical flag).
Seam: the render test sources the shipped env through the branch and
asserts the exact flag word the engine receives (render-assert, the
same pattern the spec knobs use). bash -n clean; all 22 frank-primary
seams pass.
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.
Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).
RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.
Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.
Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
Frank
