serverAppConfigs/serverAppConfigs-docs/log.md

Bundle Change Log

2026-10-18

• frank-zealot resliced to two windows of exact KV (Frank's call, PR #21). --ctx-size 786432 --parallel 3 at q8_0 K+V → 524288 / 2 at f16 K+V, so both remaining conversations keep the native 262,144 window and the compaction hub reads exact K and V. Halving the pool is what funds it: the f16 pair is 32,768 B/token over the 8 full-attention layers = 16,384.00 MiB, while the old 786,432-position pool would have needed ~24,576 MiB of KV alone on a card that leaves 24,576 addressable. Measured boot: KV 16,384.00, RS 100.50 (2 cells), compute 1,408.06 + 1,088.07, settled nvidia-smi 21,176 used / 2,949 free, n_ctx_slot = 262144, DropInPaths empty. The doc's old "f16 needs 26,112 MiB and cannot boot" figure was the 27B's 34,816 B/token applied to the 9B — corrected in the runbook.
• Rates are floors, and that is recorded as such. The endpoint had another consumer for the whole measurement window (56 prompts on the other slot in three minutes, 96 % GPU-util), so single-stream decode came back 134.1–141.5 (145.8/163.2 in a lighter minute) against the 3-slot record of 168.7, two-stream aggregate 256.6–265.1 against ≈300 across three, and prefill 3,159.5 at 106,282 beat the record's 2,877 while contended. Per-stream concurrency improved (≈130 each vs 102.8); a clean re-measure needs the box quiet.
• Instrument correction (fleet law). On the PrismML fork build 10777 the raw /completion path answers 500 "The model produced output that does not match the expected Content-only format" for a greedy ignore_eos run, so the honest per-request timings for this endpoint must be read off /v1/chat/completions. AGENTS.md said /completion; it now says the chat path.

2026-10-13

• frank-archon outage: a kernel knob, not the model. The 2026-10-12 23:17 power failure on dankserver reset /sys/module/ttm/parameters/pages_limit from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB), below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's preflight named the refusal and exited 1, and Restart=always re-ran it 6,919 times in ~20 h — activating (auto-restart), exit 1 in 50 ms, no container, no engine log. Law adopted: a sysfs raise is a countdown to the next reboot, so the gate raises the knob itself. New scripts/ttm-preflight.sh is sourced by the serve script (verify, raise, re-verify, then refuse by name) and is also the unit's ExecStartPre=+…--raise root hook — + because the service is User=dank and the knob is root-owned 0644, which is what the first attempt (a plain write inside the serve script) got wrong. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling.
• Shape: 4 × 200,000 with fp8 KV (Frank's call). --max-num-seqs 4, --kv-cache-dtype fp8, pool and slab untouched (6,912 / 22,208 MiB = the measured 29,120/card ceiling): the pool must back seqs × max-model-len, and at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12 fused-all-reduce rows, inside the 25-row bound where wht6 is bit-exact. models.yml needed no change: the per-request window is unchanged, only the concurrency.
• Deployed at bae78e7, gates green. Boot ledger: kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once — 843,276 tokens against the sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe (now reading the shape from the deployed env): four concurrent 196,960-token chats, 4/4 exact marker retrieval, 787,842 tokens in 238.4 s = 3,305 t/s aggregate prefill, no preemption; prefix cache verified (repeat answered in 1.0 s, cache_n=43008). Playwright e2e 32 passed + 1 flaky (the known Plex homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
• Re-cut twice the same evening (Frank's calls), each read off the ledger. 4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool was funding 858,969 tokens — 43 % over the contract. Then slab 24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its OWN settle floor (card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)): the real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve invented on 2026-10-11. The pair now spends 30,720 and settles unclaimed 477.30 MiB (cards ~336 MiB free).
• The prediction shipped with that arm was falsified by its own measurement, which is the whole point of banking the A/B. Claim: the slab is a THRESHOLD below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each other), so 1.6 GiB buys ~0. Measured at 25,600: 3,584 t/s single-stream cold at 43.5k (+8.9 %) and 3,589 t/s 3-way aggregate at 197k (wall 193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two points inside noise of each other are not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three slots (26,112 + 5,120 > the ceiling) and free at two.
• Capacity verified live at 3 slots: three concurrent 199,984-token prompts accepted and completed (599,952 tokens resident), ledger backs 3 sequences all at 200000 tokens at once at 607,740 tokens against the law's 608,027. A fourth full-height request is refused as too large to ever fit — radiance never truncates. Probe law: the oversized-prompt counter is FREE (the refusal fires when PROMPT tokens alone exceed the window — prompt: 202,777 prompt tokens does not fit the 200000 token context) and the window is ~514,286 characters at the measured 2.5714 chars/token, so a calibration probe below that pays a real prefill instead of reporting.
• Open call for Frank: two slots (pool 3,072 + slab 27,897 = the ceiling exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k conversation. Untested arm; capacity decision, not tuning.
• Seam: the pair ceiling is derived from the engine's budget table (30,969) and owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its own test. 371/371 tests and e2e 33/33 on both deploys.
• Power-failure casualties, NOT fixed (outside the archon window): mnt-md0.mount failed at boot, so /mnt/md0 is an empty mountpoint (the raid6 array itself is healthy, 7/7; fstab has nofail so boot continued) — media libraries on it read empty. frank-job-digest.service (frankbook job-hunt digest) is in list-units --state=failed.
• The slab is a threshold, not a slope — measured, same method (probe A leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot): 22,208 → 3,397 t/s, 24,000 → 3,290, 25,600 → 3,290. The third step bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071 (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that would clear it (pool 3,072 + slab 27,897) remains untested — and the paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots, not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
• Frank's decode challenge, answered with numbers: 512 tokens, greedy, cold — decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k (decode is nearly window-free: −6 % for 4.5× depth), while 3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate at 43.5k and 12.5 at 197k. The slot count is paid in decode concurrency — a fused-step lottery, not a fair share — which is exactly the axis the shape hunt was blind to (probe leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode table, so these rows are the baseline for this corpus, not a replacement.
• Seam: probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no future shape change can be priced on prefill alone.

2026-10-10

• Fleet service rename: the StarCraft names became the SERVICE names. The 2026-10-09 wire alias moved onto every repo artifact: units, serve scripts, env files and their FRANK_* variables, deployer service keys, container names, Prometheus/Grafana job names, bats seams, runbook filenames and all historical prose. frank-archon (dankserver :7770), frank-dragoon (dankclaw :7770), frank-zealot (dankclaw :7771). The three former service names are fully retired: git grep over the tree returns zero, rollback tags were renamed in lockstep (e.g. tag frank-archon-freetoken-pr587-20261005), and the harness models.yml lore matches. Host cutover = stop old unit + remove old container, deploy the renamed service (the deployer relinks the unit under the new name; add_pending_services auto-trashes the retired markers). The retired frank-medium/frank-small/frank-npu names are historical and stay as they are.
• Known casualty: the two vendored vLLM-fork patches under scripts/patches/ carry the old service names in context lines; the sweep rewrote them, so those patches (retired eras, rebuildable from the renamed tags) no longer apply byte-exact.
• Executed live at 55f87fc, all three gates green. Per host: stop old unit → trash its deploy-state marker + remove the unit symlink and old container → deploy.sh <host> --service <new-name>. Result: frank-archon active/enabled (container frank-archon, :7770), frank-dragoon active/enabled (container frank-dragoon-halogen, :7770), frank-zealot active/enabled (journal shows 87–93 t/s serving compaction on :7771). /health 200 ×3; /v1/models reports the three ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky, 33; Prometheus frank-archon/frank-dragoon up.
• Pre-existing gap found by the verification, NOT a rename regression: frank-zealot /metrics answers 501 — llama.cpp gates the endpoint behind --metrics (default disabled) and the serve script never passed it, so up{job="frank-secondary"} had NO series at all in the last 24 h (the zealot metrics + the queue-depth alerts on it have been dead since the Qwen3.5 swap). Fix = one flag in serve-frank-zealot.sh + a ~1 min zealot restart; not applied in the rename window.

2026-10-09

• Live — five authentik human accounts provisioned (Dankclaw, ak shell ORM, no repo objects): kirby (pk 17), ry (18), nic (19), debo (20), freya (21), each <user>@franksdank.cloud in the human-user template shape (is_active=True, password via set_password on stdin only, groups Non-Admin Users + jellyfin_users + openwebui_users). Verified through the exact PasswordStage code path (authenticate() accepts each password and rejects a wrong one) and app_entitlements(open-webui) → Open WebUI Users for all five (identical to the ridley precedent). No proxysession rows existed for them — groups were set before any login, so no claim purge was due. Downstream: Jellyfin visibility rides the LDAP outpost cache (expect lag); OpenWebUI mints their accounts as role pending on first login.

• /serverAppConfigs-docs/Infrastructure/Authentik.md — human-user template census extended with pk 17–21.

• /serverAppConfigs-docs/Dankclaw/OpenWebUI.md — new SSO bullet: first login lands as role pending (ui.default_user_role=pending in the config DB), the role-flip SQL, and the global access_grant (principal_id='*') fact that makes the picker model visible to every role='user' account.

• Live — the Protoss rename finished on the wire (all three endpoints). Served model ids moved frank-archon → frank-archon (FRANK_ARCHON_SERVED_NAME), frank-dragoon → frank-dragoon (HALOGEN_MODEL_ID), frank-zealot → frank-zealot (ALIAS). The services, units, scripts, env files and runbook pages keep the old names — container/unit identity stays stable, only the alias moved. Deploys in order: dankclaw --service frank-zealot (e2e gate 33/33 green), dankclaw --service frank-dragoon (healthy 30 s), dankserver --service frank-archon (healthy 102 s), dankclaw --service homepage. Verified on /v1/models, not on chat echoes: frank-zealot (n_ctx 200704, Q6_K), frank-dragoon (context_length 262144), frank-archon.

• Live — OpenWebUI re-keyed in the same window (snapshot webui.db.bak-20261009): model row (frank-zealot → frank-zealot, params + meta.description preserved), the access_grant row (model/frank-zealot/user/*/read), and openai.api_configs model_ids on all three backends; systemctl --user restart open-webui.service. Family-account proof, in a real browser as qa-e2e: the welcome screen selects frank-zealot and POST /api/v1/chat/completions {"model":"frank-zealot"} answers zealous with "model":"frank-zealot". A GET /api/v1/models/list probe returns {"items":[],"total":0} for that account while the picker is plainly populated — the API census is not the acceptance test, the UI snapshot is.

• Live — Hermes re-keyed (config.yaml frank-dragoon → frank-dragoon in the provider block, model.default and the three dankclaw-large auxiliary routes; backup config.yaml.bak-20261009; hermes.service + hermes-webui.service restarted, gateway log clean, /health ok) and the repo baseline dankclaw/hermes/config.yaml.example synced to the live shape (context 262144). Flagged, not fixed: Hermes still routes its dankserver-small auxiliary tasks at the id frank-medium on 192.168.0.235:7770, an endpoint that has served frank-archon since 2026-09-20 — those routes were dead before this rename and remain dead; they need a routing decision (archon capacity), not a rename.

• Deployer blind spot found by accident: every deploy.sh dankserver --service frank-archon since 2026-10-08 04:09 restarted the staging worktree, not the deploy checkout — a campaign drop-in frank-archon.service.d/bench.conf overrode Environment= and ExecStart. The health gate went green each time. Verified parity before removing it (serve script + unit byte-identical, env identical except the alias, staging HEAD an ancestor of main), moved the drop-in to /home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/, and restarted: /v1/models then answered frank-archon. Post-deploy behavioural proof is now part of the runbook.

• Repo: seams re-pinned (frank_zealot/frank_dragoon/frank_archon unit bats), fleet harnesses retargeted (bench-probe, deep-probe, calib-push, fleet-probes-swift15, autoresearch*, bench-threads-batch-http, bench-decode), shared/homepage/docs.html now teaches the ids it ships (curl/opencode/Pi send frank-zealot), AGENTS.md, OMP Harness, Homepage, OpenWebUI and the three endpoint runbooks updated. Suites: ./tests/run.sh 366 green.

• Transcript secret incident: dumping podman exec hermes-webui env printed the Porkbun API key + secret key into this session's transcript — same class as the 2026-09-16 HF-token-in-ps leak. Not committed anywhere; rotation is Frank's call.

2026-10-08

• frank-archon (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 — the radiance quantization campaign. ukisai's BF16 checkpoint (335.3 GiB, 397/397 sha256-verified) → qwen4exp-swift15-w4dn8-i8-hc8m.rad (~114 GiB): all 49,152 experts GPTQ w4 (act-order, damp 0.05) calibrated on 10,000,000 live-model tokens, 24 whole bf16-protected experts per layer derived from the calibration grams' energy share, int8 KV, fp8 HC mixtures. Engine pins: 1.0.13 for serve/convert/KLD, 1.2.0 only for calibration (the sole pin exposing RADIANCE_CALIB_TOKENS; the RADGRAM2 gram format is byte-identical across pins, so grams cross the boundary by design). Chain scripts in scripts/: make-calib-corpus.py, run-swift15-bf16.sh (bf16 reference: KLD record
  • calib tap), calib-push.py, derive-protected-experts.py, apply-protected-cast.sh, convert-swift15-w4dn8.sh, kld-gate-swift15.sh, fleet-probes-swift15.sh.
• Engine laws learned (measured, not inferred). (1) The qwen4exp_fp8 plugin declares one encoding per expert table: a recipe may NOT split ffn_gate_up_exps (w4) from ffn_down_exps (block-fp8) — the serve kernel is a single w4nl64a8h table for the whole expert set; the only legal beside-table content is whole bf16 experts (the PROTECTED section). (2) w4 expert tables stream from the file tier in place only at MNBT ≤ 2048; at MNBT 32768 the engine demands a widened form and the disk tier refuses (DID NOT FIT … stored form is not the container's). (3) The calibration tap lives in the architecture plugin (/opt/radiance/share/radiance/architectures/qwen4exp_fp8.so), not the binary — grep -ac the .so before declaring a flag missing. (4) Disk-tier boots need ~9.6 GiB of weight buffers + arena charged to the VRAM weights slab; explicit --vram-weights-mib/--vram-kv-mib pairs replaced the fp8-era EVC ratio.
• Throughput gate PASSED with 4.5× margin (campaign boot, TP=2, int8 KV, MTP 3): decode 136.0 t/s median (122–173 across probes) at 44.5k-token depth vs the 30.8 t/s bar; MTP acceptance 0.725; warm TTFT 9.45 s; prefill 6.4k t/s. The fp8-native control on 1.0.13 measured 30.08 t/s decode, and the 1.2.0 engine re-baseline was REJECTED (29.21 t/s, acceptance 0.7123 < 0.7549) — serve stays on 1.0.13.
• Deep-output audit: a 44.5k-token verbatim-recall probe enumerated all 48 planted documents in order — the DDR5-corruption signature is absent.
• Fleet probes (scripts/fleet-probes-swift15.sh): tools, reasoning levels x4, reasoning_content, response_format, vision — all PASS. KLD gate vs the BF16 reference: recorded on 1.0.13 (50,235 positions, 197 prompts), scored on the same pin at the w4-legal shape. Verdict: see runbook frank-archon-radiance-swift15-w4dn8.

2026-10-06

• frank-zealot (Dankclaw): ubatch 2048 — the second rung (llama.cpp, RTX 3090). "set ubatch to 2k" → commit d8da7a9: UBATCH_SIZE 1024 → 2048 at the 2×114,688 shape. Cost 746 MiB, not the 616 MiB the same rung measured on the MiMo shape on 2026-10-05 (a rung's price is shape-dependent; the 616 figure was not transferable). Boot: 23,120 used / 1,005 MiB free; a 512×512 vision probe then took ~240 MiB and kept it — settled 763 MiB free, the encoder's buffers staying resident across requests on this build, unlike the evening boot where the 34 MiB transient returned. Decode 52.0 t/s on the shared prompt (58.2 at the 1024 rung) with acceptance 34.2 % vs 42.8 %: ubatch does not move decode, so the gap is temp-1.0 acceptance noise, not a regression. Prefill 1,209 t/s @ 3,760 tokens — the first prefill number this endpoint has recorded, and now the baseline future rungs compare against. BATCH_SIZE stays 4096, so the seam's batch ≥ ubatch guard holds; 2048 is the ceiling for this shape (4096 = no micro-split doubles the buffer growth again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
• frank-zealot (Dankclaw): RENAMED from frank-small, then reshaped to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090). The rename (commit 87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers FRANK_ZEALOT_*/llm_zealot/deploy_llm_zealot; historical refs to the OLD FreeToken/CUDA frank-small and to the Dankserver service that became frank-medium stay frank-small, as does freetoken-frank-small.md). Live unit transition by hand before the deploy: systemctl stop frank-small + systemctl disable frank-small (a relinked unit — disable removes only its stale symlink) → deploy.sh dankclaw --service frank-zealot. The reshape (commit f2575a1, Frank's order "lower max context to 112k so we can increase ubatch to at least 1k"): CTX_SIZE 262144→229376 (per-slot 131,072→114,688), UBATCH_SIZE 512→1024 → 22,374 used / 1,751 MiB free (the per-slot cut frees ~1.05 GiB of KV; ubatch 1024 spends part, netting ~1.25 GiB back over the 489 MiB fit), decode 58.2 t/s recipe / 66.9 temp 0 (acceptance 42.8–54.4 % — the drop vs the prior boot's 63.3/72.9 is confounded by prompt text, not a shape verdict). Harness retargeted: models.yml (id/name/compactionModels, contextWindow: 114688) + config.yml modelRoles ×14. Runbook renamed: serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md.
• frank-zealot (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090). Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb): 22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), decode 48.0 t/s, vision Red. Evening: swap to ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF RCO-IQ3_XXS-mtp (10,442,827,392 B, sha256 e13d3e74…; 1.56 GiB lighter)
  • the publisher's own mmproj-BF16.gguf, MTP back on (draft-mtp depth
  3. → commit da36fa3: 23,636 used / 489 MiB free (the tightest fit in the endpoint's history — the draft ctx measured ~2.4 GiB, built against the target's full 262,144 context, not the 3-token proposal depth; the lighter quant funds it), decode 63.3 t/s recipe / 72.9 temp 0 (acceptance 49.8–62.4 % vs 48.0 no-spec), vision Red at a 128-token budget (the uncensored model thinks more — 64 tokens is all reasoning_content). ngram-mod still skipped (recurrent-state rollback law on the qwen35 hybrid). Runbook renamed + rewritten: serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md.
• frank-dragoon (Dankclaw): halogen 27B-dense → halogen-flash-server 0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770). Cutover commit 31b1ec3 (03:07:40): the v2 .hgn payload loads ONLY in the flash-server line, so the engine moved with the model. Abliterated base (refusals gone), vision back (the v2 tower loads beside the checkpoint), MTP head inside the checkpoint, 2×200,192 slots, HALOGEN_SPEC_ADAPT=0. Runbook: serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md. Incident (03:09:39–03:11:42): the deploy started the unit while host MemAvailable was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it crash-looped 8 boots, each dying in ~5 s at the pin gate, and self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9 pinned clean at 88.4 GiB available). Latent risk + fix: the host was left with only ~1.8 GiB MemAvailable, so a restart would crash-loop again; serve-frank-dragoon.sh now has a host-RAM preflight that checks MemAvailable against the pin gate BEFORE podman run and WAITS up to FRANK_DRAGOON_RAM_WAIT (600 s) for the RAM to clear, refusing loudly if it never does (the 16 GiB floor is the engine's, not HALOGEN_HOST_RESERVE_GIB).

2026-10-05

• frank-archon: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on dankserver. Built FreeToken at PR head 77a7201 on the host into the local image frank-archon-freetoken:pr587-77a7201 (digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws bought with failures: the pip-ROCm wheel needs a libhiprtc.so dev symlink for the PR's _cpu_moe link, and a stale docker login at ~/.docker/config.json poisons podman's ANONYMOUS pulls). Trio rewritten through git (env/serve/unit + seam + deployer): ONE R9700, experts in pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool 299,968 tokens (multiple-of-64 — the qsa_sparse auto backend overrides page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking → reasoning_content; models.yml flipped accordingly). Container renamed to the unit name (Frank's call; frank-archon-qwen → legacy list); deployer image gate → podman image exists (local builds carry no RepoDigests) and health gate reads the BODY — FreeToken 200s with {"status":"loading"} mid-load, which false-greened a deploy at 6 s (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK completion, reasoning split (17×24 → "408"), native tool_calls, vision "Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s single-stream (vLLM was ~30 on two cards; the KV contract caps the single-card LRU; TP=2 exists but the pynccl JIT can't compile for gfx1201 — probed, signature in the runbook). /metrics is a 404 — the Dankclaw Prometheus llm-fleet scrape of :7770 reads down until repointed. Rollback: tag frank-archon-vllm-shapeG-20261003. Runbook: serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md.
• Skill library distilled into the vault (waves 1–7). All ~173 managed skills read in full and merged, one polished reference page per skill-cluster:
  • LLM cross-cutting (16 pages, LLM/): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds, plus deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, frank-archon llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up in Model Swap).
  • Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage.
  • Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access.
  • Infrastructure (2 + updates): Observability Provisioning, Homepage Status Probes; updates to Authentik (guard-chain flip recipe, ak shell mechanics, user-provisioning playbook, outpost netns probe), Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration (filesystem/rename caveats), OpenWebUI (introspection, verification battery, ComfyUI imagegen), Hermes (config-home drift, live config edits, auxiliary routing, password rotation — the pre-existing page was preserved and extended).
  • Host pages: FrankBook-Pro ARR-Stack filled with the API-version split, schema-first probing, auth-posture method, auth-config trap, and the qBittorrent E2E verification procedure added to FrankBook-Pro/qbittorrent.
  • Desktop/Agent/Research: Noctalia and Niri (config layers, remote IPC, palette recolor, offline template render, rotation pinning, iCX3 RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center Dotfiles, index), PICO-8 Asset Standards.
  • Index updated: LLM/index.md gained a Cross-Cutting References section; Operations/ and Agent/ got index pages. AGENTS.md now points at the new sections. Link checker + full test suite green.
• e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright project. The 11-admin walk included the six *arr/qBittorrent admin links that terminate on 192.168.0.112, so every deploy gate died there when the Mac mini slept (Caddy 502 on the five vhosts, connection timeout on the raw Sonarr link). Split: those links moved to tests/e2e/specs/19-frankbook.spec.ts under a frankbook project; tests/e2e/run.sh excludes it from the default project set, and scripts/deploy.sh computes a trigger from the same BASE..HEAD range the plan uses (the spec, frankbook-pro/**, shared/homepage/services.json, the dankclaw/Caddyfile arr vhosts) and sets E2E_INCLUDE_FRANKBOOK=1. Verified live with the host asleep: full suite 33/33 in 1.1 min, the admin walk back to 5.4 s; --project=frankbook --list proves the selection. Live pass of the frankbook walk itself is deferred to the next deploy with an awake host.
  • Catch while verifying: the 17:47 Ornith-1.5-9B swap (77decb5) had left frank_zealot_unit.bats and the deployer's payload gate stale — the seam still expected MiMo-Q6K names and deploy-remote.sh gated LLM_SMALL_MODEL_FILE/LLM_SMALL_MMPROJ on files the unit no longer loads (the anti-drift seam doing its job). Both repointed to the Ornith-Q8 paths; full ./tests/run.sh back to green (366 tests).
• frank-zealot: compute-buffer pair doubled to 4096/2048, and the prefill win measured rather than assumed. Frank asked for the batch and ubatch doubled. The blocker was on record: the Q5_K_M predecessor documented "ubatch 2048 failed the scratch boot", and this shape carries 616 MiB less free space than that predecessor did.
  • The A/B used the service itself as the bench (one card, one server — no scratch port): the old rung was re-booted through a transient FRANK_SMALL_ENV_FILE systemd drop-in, and it reproduced the 2026-10-03 boot byte-identically (21,276 used / 2,849 free), which is what makes ubatch the only variable. Repo never saw a fake revert commit; scaffolding staged to ~/.AGENTTRASH/20261005-frank-zealot-ubatch-ab/.
  • At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %, for 616 MiB (free 2,849 -> 2,279). Decode flat (91.9–93.0 vs 93.8–94.3 — noise). The control that proves it is geometry and not drift: a 279-token prompt sits below both micro-batches, so it cannot see the knob, and it did not move (2,676 vs 2,658–2,811).
  • The predecessor's boot failure did not reproduce — that was a shape observation, not a law; this shape boots 2048 clean. And the fleet lesson "--ubatch-size has an optimum" is now scoped to what it actually is: true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected — the curve is backend-specific and must never be carried across backends.
  • The seam stopped hard-coding a rung: serve_argv now carries BATCH_SIZE and UBATCH_SIZE the way it already carried CTX_SIZE, and the pair test asserts the argv follows the shipped env plus the ubatch <= batch invariant. Proven live both ways — retuning the env to 8192/4096 keeps it green, inverting to 2048/4096 turns it red. (The old assertions only ever exercised the serve-script defaults, because the test env never carried these two keys.)
  • 4096/1536 is the documented middle rung; 2,279 MiB free sits at the bottom edge of the 2–3 GiB floor. Deployed e5893dc, e2e gate PASSED (2 mini specs flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a second burst and ~90 s). Docs: frank-zealot-llama-mimo-q6k-2x200704 (Compute-buffer geometry).

2026-10-03

• frank-archon: shape F then shape G — MNBT 2048, the CUDA-graph plan term pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0 proven UNBOOTABLE. Frank asked what the batch was with GPUs showing open space; the audit found the arena is budget - non-expert - KV reserve - graphs - overhead where the only deploy-reachable terms are the budget and the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB overhead are image constants; the ~3.5 GiB of driver-free VRAM was estimator slack, not allocatable).
  • Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT 4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base + 12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB / 7,653 slots, budget 31.29.
  • Shape G v1 (0a4ff30) tried util 1.0: both workers refused init before a single weight loaded — ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...), NRestarts=2 crash loop, zero payload progress. The CLI field is Field(gt=0, le=1), but the worker demands free > budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is the law, never 1.00" — now with a second proof.
  • Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to 0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES 3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
  • MEASURED live: arena 17.77 GiB / 7,762 slots (+240 over shape E), KV 401,408 tokens / 2.00x concurrency — the first shape holding two full-length conversations with real spare; capture measured 0.31/0.10 GiB against the 0.07 charge (under-charged, boots and holds); driver used 30.97 of 31.86 GiB -> 0.89 GiB real free per card, held through a live 4,096-token generation (~38-39 tok/s single-request wall-incl., +13% vs the 33.99 shape-E baseline; single sample, not a bench).
  • Verification seam rewritten to assert the shape by its OUTPUTS (util at the 0.982 law, graph charge below the analytic default and per-graph positive, KV pool >= 400,700 tokens at the measured 7,892 B/token). Runbook: serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md.

2026-10-01

• frank-archon rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4; GLM/DFlash2 declared failed and purged. The DFlash2 drafter booted and served (window 6j, 65,536 ctx, glm53-flash on :7773, both TP ranks captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with draft+hidden groups, registry method-signal), but acceptance pinned at 1.00-1.38 on every text class INCLUDING verbatim repetition — a mis-wired aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the 8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did not originate from this engine (APIServer access log: every completion POST was host-local; his workstation appears once, a /v1/models GET). Restored the verified 37.4-40.6 tok/s arm from the 2339fae trio (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter (2.2 GiB) + scratch staged to /home/dank/.AGENTTRASH/20261001-glm-purge/. KV-cache patches 02/03/04/04b survive in scripts/patches/ (generic unifier/tailpin/grouping/registry fixes, apply-verified).

2026-09-30

• frank-archon cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at the 35 tok/s target, and the shortfall is measured rather than inferred.
  • Payload Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group 128, auto_round:auto_gptq packing, ~170 GiB, 34 shards + 2 extras, 113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH expert offload: its quant_method: auto-round is overridden to inc, which routes the MoE to libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). The compressed-tensors W4A16 build of the same model (dealignai/…-CYBERSECURITY-W4A16, 182 GiB, fetched earlier) resolves to TritonWNA16Experts, which has NO offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it refused to serve at all. That checkpoint was purged with the other retired payloads.
  • Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, --enable-expert-offload --expert-precise-memory, prefix caching, vision + glm47 tool and reasoning parsers, MTP OFF, prefetch OFF.
  • Measured on the deployed engine: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
  • The wall — and a corrected one. The first account of the shortfall bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89 hit rate for 35 tok/s). A torch-profiler kernel-time window falsified it (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms/token, of which 44 % is pointwise glue — vectorized_elementwise_kernel alone 16.07 s over 708,500 launches (1,428 per token), plus 3.11 s of elementwise_kernel_manual_unroll — against 1.05 % for r4d_moe_lru_gather_kernel. The MoE GEMM is 10 %, the split-K GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor fusions that DO happen 6 %. The real cause is that the fork compiles its own Qwen trunk (@support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM trunk, so every pointwise op is a separate eager kernel; the engine says so at boot ("...provides neither a compiled submodule nor breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena 13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
  • Fork work staged (compile changelist, WIP — does not boot yet). scripts/patches/glm53-compile-01-model-decorator.patch, -02-trace-safe-probes.patch, -03-fullgraph-optout.patch, each with its own evidence header and base/patched sha256. Boots established, in order: (1) functools.lru_cache does NOT make a probe trace-safe — dynamo re-traces the wrapped body even when warm, so r4d_mhc_supported -> r4d_mhc_enabled -> available -> os.path.exists breaks the graph (a plain module-global read compiles; verified in-image); (2) suppress_errors does NOT cover graph breaks, only fullgraph=False does; (3) the GLM trunk can never satisfy fullgraph because its indexer metadata syncs to host (glm5next/amd/sparse_indexer.py:263,419), and AOT compile refuses breaks outright. With all three the trunk traces (Dynamo bytecode transform time: 1.08 s) and then dies on KeyError: 'in_proj_qkvbfg_a' in the mHC/KDA path — a latent bug the compile exposes. That is the remaining blocker.
  • No new HIP kernels are possible (the image ships r4d.so with no sources), which binary-blocks the second-largest lever: the fused router is gated num_experts in {128, 256, 512} and only r4d_moe_route_kernel<128|256|512> instantiations exist, while GLM has 288 experts. --kv-cache-dtype fp8 buys no capacity either (the plan's KV reserve is dtype-blind), only removing gfx12's only MLA decode path.
  • DFlash2 for GLM is a target-side port, not a drafter port. incoai/GLM-5.3-Flash-DFlash2 declares architectures: ["DFlash2DraftModel"] and model_type: "qwen3" (5 dense layers, sliding attention, block 8, target_layer_ids [5, 14, 24, 33, 42] of 45) and the fork already maps it (registry.py:636 -> qwen3_dflash2.DFlash2Qwen3ForCausalLM; spec method "dflash"). What glm5next lacks is the target side: aux-hidden-state capture (set_aux_hidden_state_layers / per-layer capture, as deepseek_v2.py demonstrates). No speedup is budgeted: a verification pass pays pointwise cost proportional to its token count.
  • DFlash2 tried, and the gap was exactly that one interface — now closed. Payload fetched and hash-verified (/home/dank/models/GLM-5.3-Flash-DFlash2, 2,342,169,800 B, sha256 b038e1d9… = the Hub's LFS oid). Booting with --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}' accepted the config and the drafter and died at worker init with RuntimeError: Model does not support EAGLE3 interface. The port (scripts/patches/glm53-dflash-01-aux-hidden-states.patch) took FIVE versions — hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the text model -> attrs + methods on the text model -> v5: attrs + methods on BOTH wrappers (the runner checks the LOADED instance, which for GLM is the multimodal Glm5NextForConditionalGeneration holding the text model as self.language_model; pixtral.py:498 is the reference delegation pattern). v5 CLEARS the gate: both TP workers log Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …) and the DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice by experiment-window RAM contention (see the window discipline below), so the drafter-serving end state was still unproven when this entry was written.
  • Compile work: one confirmed blocker, and a launcher bug that misled four boots. With breaks genuinely tolerated (fullgraph=False, AOT off) the trunk traces and dies on the dynamo name-map KeyError: 'in_proj_qkvbfg_a'. Four later boots reported different fatal breaks — Unsupported: Read uninitialized cell at mla.py:208 (the fused-norm JIT kernel), then Unsupported method call in the indexer's in-forward workspace acquisition (sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155 _ensure_workspace_size), then Skip calling torch.compiler.disable()'d function — and the cause was my own scratch launcher: its compile branches were restructured and the -e VLLM_COMPILE_FULLGRAPH=0 line was dropped, so those boots ran fullgraph=True, where every break is an error and the first one wins. I first blamed TORCH_LOGS=graph_breaks for the difference; that was wrong (the flag only logs). The moving reports are still useful — they are the break list — and patch 04 demonstrably removes the first of them (the fused-norm cell), patch 05 makes the workspace acquisition compiler-opaque. scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch takes GLM off the fused-norm kernel (R4D_GLM_FUSE_QKV_RMSNORM=0, a constructor flag glm5next sets at attention.py:583).
  • Diagnostics built for this (reusable). TORCH_LOGS=graph_breaks + VLLM_USE_AOT_COMPILE=0 names each break with its user stack (AOT compile hard-refuses breaks outright); /home/dank/dynamo-probe on PYTHONPATH with VLLM_DYNAMO_DUMP_ATTRS=1 prints the dynamo name-map miss detail (node target, near-miss keys, whether another live OutputGraph holds the key). Two gotchas recorded: a .pth on PYTHONPATH is NOT executed (use sitecustomize.py, chaining the image's apport hook), and the scratch launcher used to reset ENVARGS (now ${ENVARGS:-}), which silently dropped env vars and wasted a boot. All four patch artifacts are now genuine diff -u patches, verified to apply to the pinned image and to reproduce their recorded sha256.
  • Measured and rejected: MTP4 (the head loads, arch resolves to Glm5NextMTPModel, acceptance is excellent at mean length 3.76-4.90 — but decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255. The profiler window above explains it: the per-token cost is dominated by pointwise glue that scales with the token count, so verifying a draft batch pays ~N x that glue for the same N tokens of output); fp8 KV (disables the sparse MLA backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op that cannot exist on gfx12); NBATCH 8192 (boots, identical prefill).
  • Fork bug found: the plan's KV reserve under-models this model's per-sequence KDA state and the MTP draft's KV, and gpu_worker caps KV at that reserve, making a legal --max-model-len unbootable. Shipped scripts/patches/gpu_worker-kv-cap.patch (lets an explicit operator KV request raise the cap) — NOT needed by the deployed shape.
  • Rollback: commit 2339fae (qwen FP8 + MTP4); the qwen payload and the R9V state dirs are intact on disk.

2026-09-29

• frank-dragoon tuning investigation: a measured NEGATIVE result, nothing changed. Frank asked for iterative decode/prefill improvement at a 32k working point without touching the model or quantization. No win exists at gufo's served settings, so no config change was shipped. frank-dragoon was stopped for the whole investigation (~4h) per the benchmarking policy and restored healthy in 9s afterwards.
  • Measured on the repo's own prose and shell sources at 32,762 prompt tokens, salted per run so a warm prefix cache could not flatter a later run, 3–5 repetitions per configuration.
  • --prefill-chunk is dead: 512 → 8192 moves prefill 410.7–412.4 t/s. A 16x change in chunk size, 0.4% out.
  • --draft-tokens is flat: 5 → 10 gives 13.79–15.86 t/s decode, all inside the run-to-run noise band. The 5-rep confirmation round reversed the 3-rep leaders — dt9 + chunk4096 looked best and came last (12.86), while the deployed config came first (14.71).
  • --draft-policy fixed vs adaptive: no effect. --min-draft-tokens: rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for Q8 targets (Unsupported Qwen MTP tensor type for token_embd.weight: Q8_0).
  • --context 262144 → 40960 was tested as a diagnostic (the deployed value preallocates KV ~8x larger than a 32k conversation needs): no gain (13.93 vs 14.71 t/s), and it would break models.yml's declared contextWindow: 262144. Not shipped.
  • What actually moves the number is speculation itself: 6.65 t/s off → ~15 t/s on, 5.3x — already enabled. Prefill is insensitive to everything tried, consistent with gufo's own EXPERIMENTS.md where dozens of kernel variants are recorded rejected for "no release throughput improvement".
  • Two traps are now in AGENTS.md's benchmarking policy, because they cost most of the wall time and both produced false results: (1) gufo bench cannot rank speculative settings and inverted the ranking — its corpus reports acceptance=0.933 +/- 0.000 with an identical trace sha256 across draft widths (the same highly-predictable text every run, 93% of proposals accepted). It reported 29–37 t/s where real traffic gives ~15, and made dt16 look 6% faster than dt7 when real text made it 14% slower. (2) A live A/B needs >=128 generated tokens; at 8, inter_token is first-token noise and the first pass of the sweep produced a nonsense ordering that the 128-token pass reversed.
  • Also worth recording: the Server-Timing header is ttft;dur=402.485 — the unit is in the name half, so splitting on = keys the dict ttft;dur and every lookup silently returns nothing. Four configurations reported "no Server-Timing" while the header was present on all of them.
• dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s, mechanism measured. The ad hoc qwen-fp8 container (tcclaviger RDNA4 fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full flag-level A/B battery at the 32k working point; every number is from the server's usage counters (SSE chunks are invalid under spec decode). New runbook: dankserver-qwen38-fp8-expert-offload-tuning.
  • Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1, TUNABLE off — 26.80/25.15/26.81 tok/s at 166/1087/4064 ctx, acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
  • Ruled out by measurement: gather geometry 16x32 (−9%) and 32x64 (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral); prefetch depth 2 (neutral — the tier is 16 slots/layer against a ~2,400-slab working set at 73% misses); 2-way concurrency (summed 25.49 < single 26.8 — under TP both streams' misses gather on each rank, the links do not add capacity, and the shared arena thrashes).
  • TunableOp rejected: it silently skips under expert offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1; once forced on it corrupts numerics — repetition loops, the server's repetition-halt firing at ~140 of 256 tokens, "79.8 tok/s" measured on halted degenerate completions, and a fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no persistent cache in the container lifecycle.
  • Infeasible: UTIL=0.99 (free-memory wall: 31.29 of 31.86 GiB free at worker init); --max-num-seqs 2 (compile constraint violation at CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not prepared" — PLE prep is wired only for the MTP spec path).
  • The wall, measured: the link does Gen5 x8 = 31.5 GB/s/direction; the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91% of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s = 42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the fork's gather read path — fork-patch territory, not flags.
  • Telemetry fork bugs: the torch-profiler worker stop hangs on ROCm (>10 min, two attempts, while the AsyncLLM wrapper logs success in 1 s), and the default torch_profiler_with_stack=true grew a 6.5-min profile window's trace buffer ~40 GB → the kernel OOM-killed the engine core (anon-rss 95.2 GB, oom-kill in journalctl -k). The per-layer expert-cache hit rates (R4D_LRU_TELEMETRY=1) are reachable ONLY through that stop path, so the telemetry JSON is unreachable without a fork patch.
  • 256k ctx boots (the last original-goal dimension): a 262,144-token window needs 4.0 GiB of KV (hybrid attention + mamba pages ≈ 16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused (4.0 GiB needed > 2.98 available; the engine's own max-len estimate at that pool: 184,224). At KV 4.5 GiB it serves 36.65/34.21/34.77 (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just under 35). KV-exact-4.0 arm in flight for the last +3.5% slots. Correction recorded: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains.
  • Final 256k probe: the KV trim is inert. KV-exact 4.0 GiB boot-refused by 20 MB of accounting (3.98 available < 4.0 needed; engine estimate at 3.98: max len 260,176); +32 MiB (KV_BYTES= 4328521728) boots — but the arena is identical to the 4.5 GiB arm (5,648 slots): the planner buckets, the 0.47 GiB went to margin. Decode band 36.48/34.13/35.02 (acceptance 39.4%) — same 34.1-36.5 band, straddling the 35 target within noise. Final 256k config: MAXLEN=262144 KV_BYTES=4328521728, prefetch off. The full-speed alternative stays MAXLEN=184224 at 3 GiB (37.4-40.6 band).
  • GLM-5.3-Flash readiness REVISED by source reading (Frank's question about int4 vs mxfp4): the tcclaviger fork in the qwen-fp8 container registers Glm5NextForCausalLM/Glm5NextMTP with a backend-split module (glm5next/amd|common|nvidia, sparse_indexer.py for DSA); offloader/expert.py accepts R4dDsfp4MoEExperts, R4dW4a16MoEExperts, Fp8HipMoEExperts — and R4dDsfp4MoEExperts is MXFP4 weights with the DeepSeek activation law (docstring; SwiGLU gemm1_clamp_limit default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback. r4d_lib.has_dsfp4() is True in this build — native moe_dsfp4 / moe_dsfp4_mt1 entries exist, so the old "MXFP4 = emulation, missing kernels" GLM verdict is dead for this fork. WNA16 (int4) backend oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16 group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4 (~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB (> 246 GiB RAM). Feasibility doc rows updated.
  • Revert pins installed (Frank's request): (1) repo tag qwen-fp8-tuned-20260929 at f532e03 (the campaign's docs + measured config; git checkout qwen-fp8-tuned-20260929 restores the state); (2) image pinned by digest — serve script now launches docker.io/tcclaviger/vllm@sha256:da402966... (ID 3322dbbbc96d) with a local alias qwen-fp8-runtime:tuned-20260929; the floating :latest is no longer referenced anywhere (the other image on the host, 28.06.1-Code000 = fffca39f, is a DIFFERENT build — do not confuse). Runbook deployment-shape section updated.
  • Endpoint published: PORTMAP=7773:7770 hook added to the serve script; the 256k config relaunched with it. Host-side checks: health 200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists qwen3.8-flash-next, decode through the published path 36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-archon owns 7770 when it ever returns.

2026-09-28

• Dankclaw model cleanup (disk was at 93%, 137 G free). Frank asked to clean up old models. 510 G staged to ~/.AGENTTRASH/20260928-retired-models/ — moved, NOT deleted, with a MANIFEST.md giving a restore command per item. Nothing is freed until the purge is approved; that is the point of the trash policy, and the number to know before expecting the disk to move.
  • Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G, mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G, ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter 838 M. Every one maps to a runtime already retired in the runbook index.
  • The 29.3 G rollback target went with them, by Frank's decision. The payload-only rollback is now a re-fetch first; both the runbook and AGENTS.md say so. The DFlash2 drafter was NOT reclaimed — only the .gguf was taken out of that directory, because the live frank-dragoon drafter lives in it. The manifest warns against restoring the directory wholesale.
  • Kept deliberately: checkpoints/ 23 G. It is unreferenced by the repo, but ComfyUI is live and its workflows were not searched — not confirmed dead, so not taken. logs/x4e-run.log also stays: it is a sparse file, 3.3 G apparent but 4 K real, so removing it frees nothing.
  • A live reference check came first, and it earned its keep. serve-frank-zealot.sh's fallback MODEL_DIR still pointed at the retired GSQ-RCO payload frank-zealot served on 2026-09-16/17. The env file has always overridden it, so the drift was invisible — but a missing or unreadable env file would have turned a config problem into a "missing asset" naming a model that no longer exists. Repointed at Gemma4-12B-QAT/, which is what the env file already sets. Every other hit was a comment, a retired fetch script's destination, or retire_halogen_dragoon_quadlet (which moves a quadlet, not a model).
  • Not taken, by Frank's decision — scope was models only: podman images (279 G reclaimable, 95%; 170 images, 23 active) and the uv+pip caches (73 G). Together that is 352 G of regenerable garbage larger than the models reclaimed, and it is the obvious next move.
  • Pre-existing: 118 G of trash from 2026-09-21 was already staged (20260921-frank-dragoon-assembled 88 G, -redownload 30 G) and never purged. Total now staged on this host: 627 G.
  • Verified after the moves: frank-dragoon and frank-zealot both active, both /health 200, and all three live assets (Swift target, DFlash2 drafter, Gemma4 target) still present.
• frank-dragoon payload swapped to Swift-1.5 (uncensored) (production change, Dankclaw). Frank asked whether the UD Q8 of ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF would load on gufo, then said to do it. Port 7770, alias frank-dragoon, one 262,144 session, the DFlash2 drafter and the client contract are unchanged; the 27B dense weights are an abliterated (uncensored) fine-tune of the same base, under the non-OSI SWIFT OPEN LICENSE 1.0.
  • Compatibility was established from the GGUF headers, not from the filename. Both files were range-fetched and parsed: same general.architecture (qwen35), same 866 tensors with identical names in identical order, zero per-tensor ggml_type or dim mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at blk.64, same geometry. The two scary metadata differences are both no-ops, confirmed against gufo's source: general.file_type (15 vs 7) is "deliberately not consulted" by gguf_reader.cpp — the file gufo already served records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra qwen35.attention.recurrent_layers key is never read by gufo anywhere in src/; layer kinds come from full_attention_interval, which both files carry as 4 and which agrees with it.
  • Fetched and verified: 31,457,990,272 bytes, sha256 9076f99a… matching the Hub LFS oid, via the new /scripts/download-frank-dragoon-swift.sh. It sends no auth header — the repo is public and ungated, and a token on the curl command line is world-readable via /proc/<pid>/cmdline. The anonymous transfer ran at ~1.8 MiB/s; resuming with the host's token through a mode-600 curl -K config took it to ~7 MiB/s. The download survived an ssh drop by running setsid nohup, not under the session.
  • Load test, service stopped first (the GPU policy): 9.2 s from load_started to load_completed, 66.5 GiB of the 116.7 GiB iGPU at context capacity 262,144, no loader warnings. Contract battery green: context_length 262144, content + reasoning_content, finish_reason: tool_calls, SSE, and a two-turn tool loop with prior reasoning preserved. Spot-checks: "17*23" → 391; asked for a lockpicking scene it wrote one without hedging, confirming the abliteration.
  • THE TEMPLATE LAW IS GONE, by decision. Frank asked to "use our chat template"; gufo cannot take it. This is not a missing flag — src/models/qwen/chat_template.cpp is 587 lines of hand-written C++ that appends strings, with no Jinja engine, no --chat-template-file, no env override and no template body in the binary. Qwen-Sharp v22.4.0 is 453 lines of Jinja2 with namespace(), macros and raise_exception, so supplying it is a port, not a config line. Frank chose to ship on gufo's stock renderer and accept the loss: the terse system-prompt injection, in-band <|think_*|> control, the json|xml tool_call_format switch, tool-argument truncation and add_vision_id prefixes are all gone. gufo's renderer does keep the effort instruction strings and preserve_thinking. Worth watching: an abliterated model now runs without the prompt that tells it to stop waffling — the cheap fix is to send the terse text as an ordinary system message from models.yml, not to touch a template.
  • The payload is now two files in two directories. The target moved repo; the DFlash2 drafter did not move, so DRAFTER_DIR is its own tunable. That broke the deployer's stray-run sweep, which keyed on the target filename and would have silently stopped matching — a stray run would then have held 7770 past the port gate. It now keys on the drafter, which every frank-dragoon command line carries via --dflash-model and which did not move.
  • /tests/frank_dragoon_unit.bats — 20 assertions. The mutation pass found four real gaps on the first run, all now fixed: two fetch assertions were satisfied by a header comment rather than the URL assignment, so repointing a repo left them green; and nothing bound the env's MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts actually download, so a re-upload or an env edit would break the endpoint with a "missing asset" no test predicted. There is now an explicit cross-file invariant. A second harness bug worth recording: a bats --filter that matches no test reports zero not ok, which looks exactly like a mutation that survived — the harness now asserts the 1..N count before believing a result. 16/16 mutations caught.
  • Docs: runbook gained a Payload section (the header evidence), The template law section and a two-way Rollback; AGENTS.md fleet table, frank-dragoon section and rollback.
• frank-dragoon runtime swapped to gufo (production change, Dankclaw). Frank asked for github.com/gufo-org/gufo — a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151) — in place of the stock llama.cpp prebuilt. Port 7770, alias frank-dragoon, one 262,144-token session and the client contract were all unchanged; only the engine was new, and the payload was the unsloth Qwen3.8-27B UD-Q8_K_XL at that point — it was swapped for the abliterated Swift-1.5 later the same day, see the entry above.
  • The runtime is built from source on the host. gufo ships no prebuilt Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host has no Nix. Pinned at v0.1.1 / b0f8673, installed to /home/dank/GitPackages/gufo-v0.1.1/ by the new /scripts/build-gufo.sh, which is reproducible and fails closed on a moved tag. Unlike the unsloth prebuilt it replaces, gufo links the host ROCm 7.2.4 through its RPATH, so the HIP dev packages must stay.
  • Two host workarounds, both encoded in the build script. Ubuntu's librocwmma-dev ships a truncated rocWMMA header set into /usr/include that shadows AMD's complete one; the build died on 'internal/accessors.hpp' file not found, and a -I cannot fix it because in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; it is removed. Second, Ubuntu 26.04 ships /usr/include/c++/16 and the ROCm clang prefers the newest tree, whose <atomic_wait.h> needs a std::__detail::__notify_impl symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export — the link failed until --gcc-install-dir was pinned to GCC 15.
  • gufo diagnose reports [PASS] on gfx1151 (ROCm 7.2.3, amdgpu loaded, 20 CUs). It self-caps compute queues at 2 resident for serve llm after reading /sys/class/kfd/kfd/proc/*/queues/*/type, which matters because ComfyUI holds queues on the same device.
  • Shape: one session, --context 262144, DFlash2 with the adaptive block policy capped at --draft-tokens 7 (llama.cpp's fixed depth 4 is gone), greedy, --think auto --preserve-thinking on, text-only.
  • Client-contract deltas, all in the runbook: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token-prompt failure mode cannot recur); reasoning_effort now accepts the full OpenAI set, so models.yml's reasoningEffortMap fold is no longer required — kept anyway so this commit does not also move every high role from xhigh to high effort; and there is no KV-quantization or prompt-cache-RAM flag.
  • Measured at cutover: 13 s from load_started to listening (target 8 s, drafter 2 s, session 3 s) against a 20-minute LLM_WAIT budget; RSS 30.6 GiB; a live short-prompt request returned Server-Timing: ttft;dur=549.8, inter_token;dur=24.2 (~41 tok/s). Verified live: /health 200, /ready 200, /v1/models → frank-dragoon at context_length 262144, a chat completion carrying reasoning_content, a tool_calls response with finish_reason: tool_calls, and SSE streaming.
  • The trade, stated plainly. gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode +7 % at zero depth — but −14 % at 65,536 tokens of depth and −24 % at 131,072. On the 8-bit target the crossover is ~32k and frank-dragoon runs one 262,144-token conversation, so long agent sessions sit on the losing side of it. The Q4_K_XL target shows no such regression; revisiting the quantization is a separate decision.
  • /tests/frank_dragoon_unit.bats — Rewritten around behaviour: the seam runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. A caveat found while writing it: bats 1.13 ignores a failing A && B in the middle of a test (bash exempts the non-final element of an AND-list from errexit), so every such assertion was split into standalone [ ] lines. Mutations confirmed the seam bites: dead runtime gate, DRAFT_TOKENS=0, a wrong context/sessions/port, an alias swap, a moved drafter, a removed missing-asset message and a neutered commit pin each fail it.
  • /scripts/deploy-remote.sh — deploy_llm_dragoon gates on the runtime binary as well as the payload and returns before it stops the serving unit, so a host whose gufo was never built reports the failure instead of taking the endpoint down on the way there.
  • Docs: new /serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md; frank-dragoon-unsloth-qwen38-27b-q8xl.md marked RETIRED (still the rollback target — the prebuilt and the payload are both intact on disk); AGENTS.md fleet table, frank-dragoon section, bench policy and runbook index; both vault indexes; Dankclaw System Specs.md; README.md.
  • e2e gate: 32/33. The one failure is 192.168.0.112 (the FrankBook-Pro Mac mini) at 100 % packet loss — offline, and the same failure recorded in 564a781. Not related to this change; no frank-dragoon test failed.
• Frontier-MoE feasibility study on Dankserver (research + build, no production change). Frank asked which of DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box, and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run them fast. Findings, all evidenced:
  • V4.1 does not fit. 475.27 GiB checkpoint (measured from the HF API), of which 189.13 GiB is Engram in two 94.56 GiB shards. A token selects 240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a ~23 tok/s ceiling — below the 35 tok/s goal before any attention or dense work. The 1000 t/s prefill target is ~20x away: the backend refuses layer-major prefill for V4.1, so the only legal prefill is token-by-token.
  • Affinity (Codeberg, StillDeadcode) is a DeepSeek-V4-Flash engine claiming 110 tok/s on two R9700s; it builds clean here (hipcc 23, gfx1201, zero errors). The 110 works because the 101 GiB container leaves the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the quantiser, container, slab, host pool and placement engine (expert tensor naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's new indexer, the DSpark draft geometry and Engram — and Engram fails silently, because V4.1 has no num_hash_layers key so Affinity's model.h:36 default of 3 survives. Weeks of work, same capacity wall.
  • Lucebox is the only engine that runs V4.1 on gfx1201, on the unmerged feat/ds41-engine branch. Its own DS41.md measures 0.8-1.9 tok/s on an R9700 (best tuned three-tier case 9.1) with 61 GB of RAM; Dankserver's 246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is downloading for a three-arm bench.
• /serverAppConfigs-docs/LLM/Dankserver System Specs.md — RAM corrected 60 GiB -> 246 GiB (stale since the 2026-09-20 board swap; the Lucebox runbook had it right) and storage refreshed; /mnt/entropy is at 100% (244 G) and must not be planned for payload.
• /serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md — new runbook; indexed in LLM/index.md.
• LANDMINE, not changed here: /home/dank/models was cleared by Frank mid-session to free space. frank-archon is still answering :7770 from mmapped-but-unlinked inodes (/models is empty inside the running container) — the endpoint cannot be restarted until the 175 GiB payload is re-fetched. A reboot or a container restart takes frank-archon down.
• Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio (f187165 PLE 1800, 41e9189 prefix caching, ba2100e docs) was found already MERGED to main + pushed AND already live on frank-archon: the running frank-archon-r9v container (booted 12:13 from the ba2100e checkout) carries --enable-prefix-caching in argv and VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 in env, /health 200 — but deploy-state/frank-archon was still stamped f187165. A deploy.sh run would have re-selected frank-archon and reloaded the 175 GiB payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself runs on. Stamped the marker f187165 → ba2100e (status deployed, atomic mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone failure 11-admin is the FrankBook-Pro Mac mini (192.168.0.112:8989) offline (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a regression.
• scripts/frank-archon.env, runbook — prefix caching DEPLOYED on frank-archon (Frank's go-ahead; commit 41e9189): E8-qualified, zero VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200, completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is in dank's ROOTLESS podman store (sudo podman inspects an empty store and reports nothing); probe via the shim's docker or the unit MainPID.
• /scripts/frank-archon.env, /serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md, /serverAppConfigs-docs/index.md — frank-archon prefill/decode bench matrix (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT under page-cache churn (measured 740 s) — launch.sh gained R9V_PLE_READY_TIMEOUT forwarding (fork 2de0357, local-only) and prod now sets 1800.
• Incident — frank-archon down 00:06 → ~11:00 (11h): the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the R9V_PLE_READY_TIMEOUT=1800 fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own podman wait + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.

2026-09-27

• /scripts/frank-archon.env, /scripts/frank-archon.service, /scripts/serve-frank-archon.sh, /scripts/deploy-remote.sh, /tests/frank_archon_unit.bats, /serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md (new), AGENTS.md + LLM index — frank-archon cut over to the Q8_0 payload (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in R9V_CHAT_TEMPLATE_FILE/R9V_CHAT_TEMPLATE_KWARGS passthrough (fork commit 30d4a32, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-dragoon. Env points at /home/dank/r9v-state/q8xl/setup.json + the quantized-inject dev-overlay adapter (fork 363d606) + R9V_FULL_MUTABLE_CACHE=0; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's speculative_config is a single dict, no chaining, so llama.cpp's draft-mtp,ngram-mod pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
• Dankserver frank-archon slot (experiment window) — the radiance-vllm-qwen4exp fork qualified the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: runtimes/dev-overrides/qwen4_exp.py dequantizes quantized block_inject shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and R9V_FULL_MUTABLE_CACHE=0 passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (mapping_bytes=54400261120, mapping_rss_bytes == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt; record at fork docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md.
• Dankserver frank-archon restore — uncensored UD-IQ4_XS back on 0.0.0.0:7770 as frank-archon-r9v from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); frank-archon.service (llama.cpp) runtime-masked for the window (/run/systemd/system/frank-archon.service → /dev/null — the /etc relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs restart=no, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env; scratch in dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/.

2026-09-25

• /serverAppConfigs-docs/Dankserver/Jellyfin.md — Jellyfin LDAP 401 resolved: the auth works via the modern /Users/AuthenticateByName endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy /Users/Auth/Password route, which Jellyfin 12.1 removed (the LegacyEmbyRouteRewriteMiddleware is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
• /shared/homepage/services.json, /shared/homepage/docs.html — Suggested uses added to the three LLM model descriptions (status cards + the /docs model guide + the "which one to pick" foot-note): frank-dragoon → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-zealot → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to /srv/homepage; bats 19/19 + non-Mac-mini e2e green (d3c8197).
• /shared/homepage/{services.json,app.js,style.css,docs.html} — Model cards reworked to a structured three-line shape: each LLM card now carries a details object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (app.js descHTML() falls back to the plain description for non-LLM cards; new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset). Roles made neutral (no nicknames): frank-dragoon = deep reasoning, frank-medium = primary model, frank-zealot = fast utility model. The /docs model guide was also fixed — it had drifted to retired model info (frank-dragoon listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
• Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik 2026.8.0→2026.8.3 (server/worker/outpost), tautulli v2.18.1-ls242→ls245, calibre-web 0.6.27-ls397→ls402, plex 1.43.3.10896→1.43.4.10903; seerr already at v3.4.1. Local-build rebuilds (forced --no-cache, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh :latest, 23986f31→e1e87aec). anubis already at latest stable v1.27.0 (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
• /dankserver/{tautulli,calibre-web,plex}/quadlet/*.container, /dankclaw/authentik/quadlet/*.container — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead dankserver/compose.yaml plex entry synced (0b1ea2e).

2026-09-22

• /serverAppConfigs-docs/** — OKF v0.1 frontmatter migration: type/description/tags/timestamp added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); timestamp added to LLM/frank-medium-batch-ubatch.md and LLM/frank-zealot-benchmark.md; unquoted YAML scalars (colons, *arr) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved LLM/Pipeline Parallelism/index.md; log.md restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
• /serverAppConfigs-docs/** — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in Shared Memory Architecture Dankclaw.md, deployment.md + Architecture/Deployment Flow.md rewritten to the Quadlet/deploy-plan.py truth, Calibre.md 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), Copyparty.md v1.20.24, RomM.md 5.3.0 breaking change; *arr docs moved to a new FrankBook-Pro/ section (git mv, link-safe); active runbook bodies written for frank-dragoon (unsloth UD-Q8_K_XL) and frank-zealot (Gemma4-12B).
• /serverAppConfigs-docs/Research/ — Three new research briefs (OKF): Local TTS and Voice Cloning.md, AI Job Search and Resume Tools.md, Most Popular Self-Hosted AI Apps.md (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report popular-self-hosted-ai-apps.html; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + /api/v1/images/generations, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.

2026-09-21

• /dankserver/frank-medium/quadlet/frank-medium.container — frank-medium cut back to the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq); KV_DTYPE became a quadlet tunable, set auto (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
• /dankserver/calibre-server/quadlet/calibre-server.container — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, --auth-mode=basic pinned, internal port 8090.
• /serverAppConfigs-docs/Dankserver/Calibre.md — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
• /scripts/serve-frank-dragoon.sh, /scripts/frank-dragoon.env, /scripts/frank-dragoon.service, /scripts/download-frank-dragoon-unsloth.sh — frank-dragoon (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB), 200k ctx, MTP depth 2 via explicit -md; the split parts ARE the payload — concatenating them trips invalid split file name (part 1's header advertises file_count=3); draft acceptance verified live 0.83–0.92.
• /serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md — New runbook; frank-dragoon-ciru-orca.md marked RETIRED (rollback target); AGENTS.md + LLM index updated.
• /shared/prometheus/prometheus.yml — node-dankclaw target fixed to host.containers.internal:9100 (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); Infrastructure/Prometheus.md targets table + tests/observability_quadlet.bats contract updated.

2026-09-20

• /dankserver/frank-medium/quadlet/frank-medium.container — Checkpoint swapped MXFP4 → FP8 (orcarouter/Qwen3.8-27B-Uncensored-FP8, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via scripts/hf-manifest.py + scripts/hf-fetch.sh + scripts/swap-frank-medium.sh; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
• /dankserver/frank-medium/quadlet/frank-medium.container — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to scripts/.AGENTTRASH/20260920-llama-layersplit/ (never systemctl disable); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
• /scripts/build-llama-rdna4.sh — Now takes GGML_HIP_RCCL; the RCCL build (build-rdna4-rocm714-rccl) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — invalid device function).
• /shared/prometheus/rules/llm-fleet.yml, /shared/prometheus/prometheus.yml, /shared/grafana/provisioning/ — Monitoring spans the vllm: + llamacpp: namespaces (6 alerts + 3 recording rules); frank-zealot/frank-dragoon targets fixed to host.containers.internal; the prefix-cache reuse rule divides by cached + prompt (llama.cpp's prompt_tokens_total EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
• /scripts/dankserver-route-primary.sh — eno1 pinned primary (metric 100), eno2 retained failover (metric 200); applied with nmcli device reapply (no connection reapply in NM 1.54).
• Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge AtomicOpsCap: Routing+, both endpoints ReqEn+ — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
• /serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); frank-medium-llamacpp-dflash2.md carries the SUPERSEDED banner; Dual-R9700-Cooperation.md states the measured verdict; Dankserver System Specs.md gained a Motherboard section; frank-medium-model-swap.md is the new swap runbook; AGENTS.md topology/sections/lessons flipped.

2026-09-19

• /scripts/frank-dragoon.service, /scripts/serve-frank-dragoon.sh — frank-dragoon: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
• /serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md — New runbook.
• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
• /scripts/frank-zealot.env, /scripts/serve-frank-zealot.sh — frank-zealot: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
• Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
• AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.

2026-09-18

• /scripts/frank-medium.env — CACHE_RAM 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; tests/frank_medium_unit.bats seam added; --ctx-checkpoints 0 confirmed a deliberate deviation (build default is 32) (1acca43).
• Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned llama-server from third_party/llamacpp/build-merge (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.

2026-09-17

• /scripts/frank-zealot.env, /scripts/serve-frank-zealot.sh — frank-zealot: ISTA-DASLab GSQ-RCO IQ3_S → HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M + separate MTP assistant sidecar (--spec-type draft-mtp,ngram-mod); q8_0 KV (build-cuda rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON, else ~12× prefill collapse); --fit chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every reasoning_effort level.
• /scripts/frank-zealot.env — CTX_SIZE=1048576 / PARALLEL=4 (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
• /serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md — New ACTIVE runbook; frank-zealot-qwen38-gsq-rco.md retired as rollback target.
• /shared/homepage/ — The three LLM cards moved to the user page with live descriptions; new static /docs user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
• /scripts/deploy-remote.sh — The pending pass now prunes stale pending state entries for retired services into ~/.AGENTTRASH and keeps going (a stale frank-default-sightsound entry made every --pending run exit 2) (02a7876).
• Fleet audit — Every vhost classified against the guard policy (no new exposure); systemd-hermes logged as a boot-persistence gap (running container, no quadlet); stray stopped test-vllm container on dankserver held for approval.

2026-09-16

• /scripts/bench-batch-ubatch.sh — New harness: one llama-bench invocation per cell (a test that cannot allocate exit(1)s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (LimitMEMLOCK=infinity is load-bearing) (87845fc, 77a07ff).
• /scripts/frank-medium.env — UBATCH_SIZE 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and TENSOR_SPLIT 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); tests/frank_medium_unit.bats balance seam re-fitted as a shape assertion.
• /serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
• /dankclaw/frank-dragoon/quadlet/frank-dragoon.container — halogen restored as frank-dragoon, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-zealot); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
• /scripts/frank-zealot.env, /scripts/frank-zealot.service, /scripts/serve-frank-zealot.sh — New frank-zealot: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (draft-mtp), BF16 projector, 131072 ctx, f16 KV, port 7771.
• /scripts/deploy-remote.sh — deploy_llm_dragoon flipped back to halogen with a wait_port_free 7770 gate (the first cutover died on Address already in use two seconds after the old server exited); deploy_llm_small added.
• /serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md, /serverAppConfigs-docs/LLM/halogen-flash-server.md — New / re-activated runbooks; frank-dragoon-ik-uncensored.md marked retired.
• /serverAppConfigs-docs/LLM/x4-pair-experiment.md — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by -ot itself (has_tensor_overrides() → pipeline_parallel → n_copies == 1); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (-dev A,B is a SWEEP, not a list — correct -dev A/B -ts x/y gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
• Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every podman command failed, all 28 quadlets looped, podman system migrate panics in 5.7.0; a reboot repaired it. The reboot exposed hermes.franksdank.cloud 502: the Caddyfile dialed 192.168.0.143:8787 (a host bind, not a published port) — fixed to host.containers.internal:8787, and the seam that had accepted any numeric upstream was strengthened.
• Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; halogen-flash-server.md gained a "Refreshing the weights" note.

2026-09-15

• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh, /scripts/frank-medium.service — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (retire_root_llm_small() archived the CUDA-era unit).
• /serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md — New runbook; freetoken-frank-small.md and kingjones-q8agent-rocmfpx.md marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
• Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the ld.so.conf hijack (staged in ~/.AGENTTRASH/20260915-rocm633-cleanup/).
• /scripts/frank-dragoon.service, /scripts/serve-frank-dragoon.sh — frank-dragoon cut to ik_llama.cpp (halogen quadlet retired to ~/.AGENTTRASH/20260915-ik-swap/); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
• /serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md — New runbook with the full evidence chain.

2026-09-14

• /dankclaw/compose.yaml, /dankclaw/.env.example — New hermes-webui service (Hermex): bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data, loopback-only publish 127.0.0.1:8787, password-gated via HERMES_WEBUI_PASSWORD.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories.
• /tests/hermes_webui.bats — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
• /serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md — New page; remote access = Termius local forward over the existing router :22 port forward.

2026-09-12

• /dankclaw/marreta/quadlet/ — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; NGINX_HTTP_PORT=8085 (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
• /tests/marreta_quadlet.bats — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
• /serverAppConfigs-docs/Dankclaw/Marreta.md — New page (incl. the base64:-prefixed APP_KEY gotcha).
• Live — authentik ProxyProvider marreta (pk 13) + Application link: the outpost's config API filters application__isnull=False, so a provider with no linked Application is never loaded (the login-wall root cause); a null invalidation_flow crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec 16-marreta.spec.ts.

2026-09-04

• /frankbook-pro/ — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, /opt config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed 10.88.0.1:8080 → 127.0.0.1:8080.
• /tests/frankbook_quadlet.bats — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
• /dankserver/ — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned /opt/<svc>/config dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and userns_mode must NEVER be added (podman-compose runs services in pods; --userns + --pod is rejected).
• /dankclaw/Caddyfile, /dankclaw/pihole/addn-hosts — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
• /tests/e2e/ — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).

2026-09-03

• /serverAppConfigs-docs/ — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, Prowlarr.md/prowlarr.md deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.

2026-09-01

• /frankbook-pro/gluetun/, /frankbook-pro/qbittorrent/ — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (WebUI\Password_PBKDF2, not the legacy WebUI\Password key qB ≥ 4.2 ignores); DefaultSavePath → /Volumes/COMPLETE/Finished Torrents; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
• /serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
• /dankclaw/authentik/quadlet/authentik-outpost-ldap.container — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via ak shell ORM; live bind tests proved two rules: the token intent must be verification, and the provider needs the search_full_directory object permission.
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Dankclaw/Jellyfin.md — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (LdapSearchAttributes is comma-separated; username attribute cn).
• /scripts/frank-zealot.env, /scripts/serve-frank-zealot.sh — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in _dequant_fp8_weight); reverted to Huihui FP8 within the hour; freetoken-frank-small.md quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
• /dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/ — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move /opt/<svc> → /home/dank/Containers/<svc>); v1 generator quirks fixed (Command=→Exec=, Device=→AddDevice=, HealthCmd= space-splitting, the HOME=/root EnvironmentFile trap).
• /dankclaw/ — Data roots of gitea, authentik, and the hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime); deploy_hermes ordering fixed (generated units reload AFTER the unit files are copied).
• /dankclaw/seerr/quadlet/seerr.container — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider seerr-proxy; /tests/seerr_quadlet.bats new seam; /serverAppConfigs-docs/Dankclaw/Seerr.md new page.
• /scripts/deploy-remote.sh — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + HostHeaderValidation=false + the four categories; full 7-service deploy run.
• /frankbook-pro/qbittorrent/quadlet/qbittorrent.container — qB pinned 4.6.7 (5.x's 204 login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
• /serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md, /serverAppConfigs-docs/Architecture/Storage Policy.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.

2026-08-30

• /scripts/frank-npu.env, /scripts/serve-frank-npu.sh, /scripts/frank-npu.service — New frank-npu on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (/dev/accel/accel0 err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with iommu=pt + reboot; vision + audio towers proven end-to-end; health probe is /v1/version (FLM has no /health route) (c3ed3a6, cf0c541).
• /serverAppConfigs-docs/LLM/frank-npu.md — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
• /dankclaw/frank-dragoon/quadlet/frank-dragoon.container — frank-dragoon: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the pinned kingjones30/ROCmFPX fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
• /scripts/frank-zealot.env — frank-zealot (FreeToken): model swapped to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8); ~40 tok/s verified.
• /serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md — Reactivated as the large runbook; agentionai-fast-imatrix.md marked RETIRED; freetoken-frank-small.md model sections updated; LLM index + AGENTS fleet tables refreshed.
• /serverAppConfigs-docs/Dankserver/prowlarr.md, /serverAppConfigs-docs/Dankserver/qbittorrent.md — Rewritten: the arr↔Prowlarr feed contract (/{indexerId}/api, the global key — the hand-rolled /torznab/ paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (WebUI\Password_PBKDF2, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).

2026-08-29

• /dankclaw/Caddyfile — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
• /dankclaw/anubis/policy.json — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile).
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Infrastructure/Anubis.md, /serverAppConfigs-docs/Architecture/Network Architecture.md — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original Host vs each provider's external_host (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null authorization_flow/invalidation_flow starve the outpost's config fetch; the public zone gained the auth A record via the Porkbun API (ee1e038).

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
• /dankclaw/gitea/quadlet/gitea-caddy.container — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
• Incident — Post-reboot: Caddy (a bare compose container — restart: unless-stopped does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: net.ipv4.ip_unprivileged_port_start reset to 80 by a later-sorted sysctl drop-in AND a stale line in /etc/sysctl.conf, which systemd-sysctl applies AFTER all of sysctl.d/.
• /AGENTS.md — New MANDATORY "Boot Persistence & Self-Healing" section.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (joins dankclaw_default, Pull=never, env from /home/dank/Containers/caddy/env); removed from compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — Repo source of truth for ip_unprivileged_port_start=53; deploy_pihole stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies sysctl --system (the boot code path).
• /tests/caddy_quadlet.bats (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
• /serverAppConfigs-docs/Infrastructure/Caddy.md, /serverAppConfigs-docs/Infrastructure/Pi-hole.md, /serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md, /serverAppConfigs-docs/Dankclaw/Gitea.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.

2026-10-01 — frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)

bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without --expert-precise-memory:

• with precise: NRestarts 1->3, each attempt ~6.7 min then Deactivated successfully (API server drains after an EngineCore fatal error) — cause not captured, the serve script rm'd the container.
• without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts climbing, container Exited (1), and the journal finally showed the real error: RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0] as dynamic but your code specialized it to be a constant (3)' — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM. ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem, so a stale compile cache is ruled out: the conflict is produced by this configuration's own compile path.

Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048, KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger; --enforce-eager is the bypass that would sidestep the constraint path.

Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag. Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).

2026-10-01 — frank-archon: --expert-precise-memory + bf16 = boot crash loop

Reverted. The two changes Frank asked for were applied together and the result crash-looped: NRestarts climbed 1 -> 2 -> 3, each attempt running ~6.7 minutes then logging Deactivated successfully (vLLM's API server drains and exits 0 after an EngineCore fatal error, so podman wait returns 0; the new Restart=always turned a silent death into a visible loop).

The budget arithmetic explains it without needing the error text, because the arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and that is exactly what rocm-smi measured):

line	bf16 2 x 262144 + precise	fp8 3 x 262144
non-expert weights	6.28	6.28
pinned pool	6.80	5.68
CUDA graphs	0.55	0.55
arena	16.65 (precise: overhead 5.25)	14.31 (default: overhead 7.60)
non-torch	~1.45	~1.45
total	31.73 of 31.86 GiB	28.27 (measured)
free at boot	~0.13 GiB	3.59

--expert-precise-memory hands the 2.35 GiB compile/graph-capture headroom to the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they put graph capture ~0.13 GiB from the ceiling, where every transient allocation dies. So the flag is dropped and bf16 2 x 262144 ships with the default overhead: arena 14.31 GiB, ~2.4 GiB free at boot.

Lesson: the registry's "compile headroom" is not slack to be reclaimed by default — it is the budget for the phase that allocates last. Reclaim it only by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).

2026-10-01 — frank-archon: 2 bf16 windows + --expert-precise-memory

Frank's final shape for the night. Two changes at once, both measured in principle before the boot:

• KV is bf16 (--kv-cache-dtype auto), two full-size windows (2 × 262144). bf16 measures 13,839 B/token (from the bf16 3×241664 boot: 11,047,010,304 B -> 798,280 tokens), decomposing as 12,276 attention + 1,563 mamba/side caches against fp8's 6,138 + 1,563 = 7,701. So 524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
• --expert-precise-memory. The registry's default overhead for this architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom. The flag selects the fitted figure, which moves only the plan's room term — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the cost of boot-time capture slack. The overhead formula adds 0.22 GiB per 1024 nbt above 4096, so MNBT 2048 pays nothing.

Budget note worth keeping: the arena is (budget - non_expert_weights - kv_reserve - graphs - overhead) // expert_bytes, where kv_reserve is the planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale with SEQS. The pinned --kv-cache-memory-bytes therefore does not shrink the planned arena; it comes out of free VRAM instead.

2026-10-01 — frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions

Frank's call. Payload swapped from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB) to unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB, sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer uncensored). New fetcher scripts/download-frank-dragoon-q4.sh; the DFlash2 drafter and its fetcher are unchanged.

Shape: --sessions 4 --context 163840 with unquantized KV. gufo has no KV-quantization lever at all (execution_policy.hpp pins kv_cache_storage to kFp16), so "unquantized" is inherent.

KV arithmetic (why this fits where one 262144 session used to sit): f16 KV costs 64 KiB/token — 16 full-attention layers (the hybrid is full_attention_interval: 4 over 64 layers) x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 = 655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget: weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB against the 114 GiB GTT.

2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner

specs/12-crawl.spec.ts:23 ("every user open link reaches its app") failed the deploy gate twice and then passed cleanly on a manual re-run:

• run 1 (deploy): Expected "plex.franksdank.cloud" / Received "app.plex.tv" — Plex redirects its web UI to the hosted app; the same failure appeared on the previous deploy too.
• run 2 (manual): page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/ — the homepage answered http=200 in 40 ms from the dev box at the same moment, so this is a browser-level flake.
• run 3 (manual): 33 passed, exit 0.

So a red gate on this spec alone is NOT proof the fleet is broken, and the deployer's DEPLOY INCOMPLETE banner can be raised by a flake. Re-run tests/e2e/run.sh before treating it as real; the Plex redirect is the one stable underlying issue and it lives in Plex's web-client/image config, not in any repo service.

2026-10-01 — frank-archon bf16 no-spec cutover

• /scripts/frank-archon.env — bf16 no-spec cutover, FINAL shape after the night ladder: MAXLEN 241664 x SEQS 3, KV_BYTES 10,184,000,000/rank, KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner: "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per request: 3.03x".
• The night ladder (Frank's method). 262144 OOMed at CUDA graph capture (Tried to allocate 76.00 MiB ... 0 bytes free, 30.43 GiB of 31.86 GiB allocated by PyTorch) and 245760 OOMed identically; 237568 PASSED (KV 718,606 tokens) and the +4096 bonus step 241664 PASSED and is PRODUCTION. 241,664 is max possible on this box at util 0.98 — the model's native 262,144 does not fit with a 3-seq contract. Every rung was booted under a transient systemd unit at uid dank with LimitMEMLOCK=infinity (the rootless bench probe masked the production memory environment) with full engine-log capture.
• Util stays 0.98. The earlier 0.90 edit was wrong — lowering --gpu-memory-utilization shrinks the engine's own budget rather than buying slack; Frank forbade it and the winning ladder ran at 0.98.
• /scripts/serve-frank-archon.sh — --speculative-config line removed; --max-num-batched-tokens and the -e R4D_LRU_GATHER_CHUNKS/LANES container-env passthrough added; shape header rewritten.
• /tests/frank_archon_unit.bats — seam re-pinned: bf16-KV-present, spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the qualified-shape and flags tests flipped to the new env keys.
• Overnight redo campaign (prefetch OFF everywhere) tabled in /serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat 1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone, six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald -o json).
• /AGENTS.md — fleet table frank-archon row rewritten to the bf16 shape.
• /serverAppConfigs-docs/LLM/index.md — fleet intro + mermaid node updated out of the GLM era.
• Client contract fixed (thinking + tools). The live endpoint returned a null reasoning field and dumped the thinking text INLINE in content with a dangling </think> (measured: 'We need answer…Brief.\n</think>\n\n391') — the malformed thinking block clients choke on — because the deployed argv had no --reasoning-parser. The GLM rollback (d91379d) had dropped all three contract flags and the qwen rollback inherited the gap. Restored from ecaad36's values, confirmed against this fork's registries: --reasoning-parser qwen3, --enable-auto-tool-choice --tool-call-parser qwen3_xml, --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}'. The seam's "NO tool dialect" test is flipped to a PRESENT test.
• Final shape (Frank's call): 5 seqs x the full native 262144 window at fp8 KV. 6 x 241664 at fp8 died (Tried to allocate 10.70 GiB, 9.18 GiB free) — six full-length contexts do not fit at any KV dtype — so the contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first rung, KV_BYTES 10,388,766,720/rank, measured 1,350,182 tokens / 5.15x concurrency (39,462 tokens of margin). Seam re-pinned; the bf16 three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the same night) is the documented max at 3 contexts.
• Harness /home/blank/.omp/agent/models.yml — contextWindow back to 262144 with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365 <= 262,144) and preserve_thinking moved from a bare top-level body field to chat_template_kwargs (the form vLLM actually reads).
• Runtime-survivability correction: five slots -> FOUR. 5 x 262144 fp8 passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free at runtime; a real client request (995-token prompt, max_tokens=261149, sampling params straight from models.yml) killed Worker_TP1 with torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free and the engine shut down. Boot is not proof of runtime survivability — after a knife-edge shape change, send a large-max_tokens request before calling it done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per rank for activations/graph capture. Frank's call 2026-10-01.
• /scripts/frank-archon.service — Restart=always, not on-failure (plus StartLimitIntervalSec=0 and OOMScoreAdjust=-500). A runtime worker fault makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed down until a human noticed. An explicit systemctl stop still wins, so bench windows and deploys are unaffected.
• Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, MNBT 2048, rest to the arena. MAXLEN 262144 x SEQS 3 = 786,432 tokens of contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
• DCP investigated and closed (see the parallelism survey in the runbook): --decode-context-parallel-size 2 is rejected at boot — DCP for GQA requires TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning knob on this box.
• Production bring-up: the shape boots and serves on :7770 — engine log verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live contract probes (reasoning split, tool_calls). Rollback target for the 2026-09-29 fp8+MTP4 shape: git tag frank-archon-fp8-mtp-20261001.

2026-10-01 — frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP

• Question that started it: can a Qwen3.5 MTP drafter attach to frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint and MiMo ships ZERO mtp.* tensors (base and INT8 alike). Answer 2: the digest-pinned kyuz0/vllm-therock-gfx1151 image (vLLM 0.30 / torch 2.11 / TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and vLLM reads the draft from a SEPARATE path — so cross-model MTP from Qwen/Qwen3.5-9B (the distill's source, identical 4096/248320/32-layer geometry) works. Frank's call: ship that as frank-dragoon.
• Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token prefill / 512 decode): base 17,876 t/s prefill / 13.1 t/s decode (13.1 is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / 19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec arms pay ~55-59% of prefill. num_speculative_tokens beyond 2 unmeasured.
• Measured numerics law: spec arms do NOT reproduce the no-spec greedy text (bf16 batched verification reorders reductions; near-tie argmaxes cascade). vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
• Client-contract delta wired: vLLM emits reasoning, not reasoning_content - models.yml frank-dragoon block now sets reasoningContentField: "reasoning", and the extraBody moved to the Qwen3.5-family recipe (0.7/0.8) with chat_template_kwargs (a top-level preserve_thinking is ignored by vLLM).
• Deployer gap found pre-deploy: the spec-draft gate demanded a single model.safetensors (the dflash layout) and would have rejected the deployed MTP arm (4 shards behind an index); fixed to accept both layouts, gated by bats. z-lab/Qwen3.5-9B-DFlash's "vLLM PR #40898 unmerged" card claim is FALSE - vllm/v1/spec_decode/dflash.py ships in 0.30 (verified in-image); the drafter stays on disk as the alternative.
• Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB, DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo rollback pinned at git tag frank-dragoon-gufo-q4kxl-20261001; engine, Q4_K_XL payload and DFlash2 drafter all remain on the host.
• First deploy attempt #2 crash-looped the container 51 times: the serve script handed vLLM the HOST model path while the tree is mounted at /models inside the container — transformers rejected it as a malformed repo id. Fixed by converting to the container path (the spec draft was already converted; the target was not), pinned by bats. Also fixed the deployer's own draft gate (same single-file bug as the serve script's).
• Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via scripts/download-frank-dragoon-swift.sh) and the orphaned flux-2-klein tree under models/checkpoints (23 G; ComfyUI mounts comfyui-data/models, not this) purged via ~/.AGENTTRASH/20261001-frank-dragoon-reclaim/ — 52 G freed. KEPT the 1.1 G Qwen3.8-27B-DFlash2-Q4_K_M.gguf (it is the gufo rollback's drafter, not a remnant).

2026-10-02 — frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP

Frank's order was the highest quality of kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF on one full context, "whatever runtime it requires". The answer is forced by the file: Q8_0_ROCMFPX_AGENT.gguf carries tensor type 103 (GGML_TYPE_Q8_0_ROCMFPX, ftype 115), which stock llama.cpp, gufo and every other loader on the fleet cannot read — only the kingjones30/ROCmFPX fork can, so the runtime is pinned to commit 36e9acd40e10a87cd3c3ef8ec734668757dc8520 and built by scripts/build-rocmfpx.sh. Shape: 1 session x 262,144 (native), f16 KV, the bundled BF16 mmproj, and MTP from the repo's own mtp-Qwen3.8-27B-Q4_0.gguf (n-max 4, --spec-draft-device ROCm0).

• MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without (+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 / mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT routing variant was chosen over the card's plain Q8_0 because it keeps 187 output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance 0.953 vs 0.911.
• Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
• Rollback: git tag frank-dragoon-gufo-q4kxl-20261001. The retired vLLM-era fetchers moved to old/. Runbook frank-dragoon-rocmfpx-q8agent-mtp.

2026-10-02 — frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots

Three orders in one day on the same endpoint: bartowski/…Q5_K_M on llama.cpp with two contexts ("whatever context size fits the 3090"), then three contexts "as much as can fit while keeping overhead open space to 2-3gb on the 3090", then "frank-zealot should have vision".

• Final shape: CTX_SIZE=540672 PARALLEL=3 = 3 x 180,224, f16 K + q8_0 V under flash attention, --mmproj vision, all layers on the 3090. Measured 21,684 MiB used / 2,441 MiB free of 24,576 — inside the 2-3 GiB band.
• The pool came from two boots, not a formula. 2 x 262,144 with no projector: 20,190 MiB used, KV 12,800 => non-KV 7,390 MiB. 3 x 196,608 with the projector: 22,972 used, KV 14,113 => non-KV 8,859 MiB. The difference prices the vision tower at ~1.4 GiB for a 918 MB file — the encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608) booted healthy but left only 1,153 MiB free, which is how the band got violated and why the seam now prices non-KV at the measured 9 GiB and REJECTS that shape.
• Three contexts cost window, and that is the tradeoff Frank was told: 180,224 tokens per conversation, not the native 262,144. A longer prompt is a hard 400 exceed_context_size_error, so models.yml sets contextWindow: 180224. Consequence at the time: frank-archon's then 183,501-token compaction request did not fit here with summary room — resolved later the same day when frank-archon's window moved to 200,704 and its trigger fell to 140,492 (see the shape-D entry below).
• Measured (2026-10-02, this shape): decode 106.8 t/s, prefill 3,814 t/s (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention scan), three concurrent requests answered in parallel in 1.7 s with a fourth queued at 3.0 s, native tool_calls, reasoning_content populated, reasoning_effort: low accepted, solid-red PNG answered "Red."
• Vision contract gotcha: /v1/chat/completions ignores the legacy top-level images field — that request answered "Brown." for the same red image, i.e. it generated from text alone. Only the image_url content part reaches the model. /completion accepts images but this build still requires messages.
• Deployer gates on the projector too (LLM_SMALL_MMPROJ), so a missing mmproj cannot silently ship a blind endpoint; MMPROJ_FILE= (explicitly empty) in the env file is the documented text-only opt-out and returns ~1.4 GiB to KV.
• Reclaim staged, not purged, awaiting approval: ~/.AGENTTRASH/20261002-retired-llm-payloads/ (~59 GiB of superseded payloads). Kept in place for rollback: MiMo-Q6K/, MiMo-Q5KM/, Qwen3.8-27B-UD-Q4_K_XL/ and Qwen3.8-27B-UD-Q8_K_XL/ (the latter holds the gufo DFlash2 drafter). Runbook frank-zealot-llama-mimo-q5km.

2026-10-02 — frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts

Frank's order: shrink frank-archon to a 200704-token window with about two resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The first attempt (30904ba, 2 x 200704 + SEQS=2) crash-looped; the revert went live as e460bbb, and the shape was then re-applied one law at a time on top of the proven-good base, root cause first.

• ROOT CAUSE: --max-num-seqs 2 cannot compile this model. It was never OOM, never MNBT, never the KV dtype. Four single-variable boot probes against the then-deployed shape (production stopped for each, candidate booted AS the service):

  MAXLEN	SEQS	MNBT	result
  262144	2	1024	Engine core initialization failed
  262144	2	2048	Engine core initialization failed
  262144	3	1024	healthy - 792,069 tokens, 3.02x, arena 6,248
  262144	3	2048	healthy - the then-deployed incumbent

  torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant (3), raised from AOT build_guards(strict_error=True) during determine_available_memory -> profile_run -> _dummy_run, ~7 min in. model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets num_reqs = min(max_num_batched_tokens, max_num_reqs), so its width is SEQS + 1 for ANY MNBT >= SEQS - 3 at SEQS 2, 4 at SEQS 3, exactly the constant in the message. MNBT only moves the compile RANGE ((1,1024) vs (1,2048)), which is why the identical error appears at both rungs. --enforce-eager bypasses the path but drops CUDA graphs. The 2026-10-01 bf16 2 x 262144 failure was this same error class; its message named "KV dtype and SEQS" together and nobody bisected, so the dtype was blamed - it was a red herring.

• THE ARENA LEVER IS max_model_len, NOT --kv-cache-memory-bytes. The planner sizes expert cache as budget - non-expert - KV reserve - graphs - overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len, independent of the pool override:

  MAXLEN	KV reserve	arena
  262144	2.48 GiB	14.30 GiB / 6,248 slots
  200704	1.91 GiB	14.87 GiB / 6,496 slots

  d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 pool cut (2.75 GiB/rank) bought zero arena slots. The previous session's premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool cut is a runtime-HEADROOM lever (the shape it replaced died with 0 bytes free on a real 995-token request).

• Final shape (option D): --max-model-len 200704 x 3 slots, fp8 KV, --kv-cache-memory-bytes 3,150,000,000/rank, MNBT 1024, gather 32x64, no speculative decode, prefetch off, no --expert-precise-memory. The argv keeps the only slot count that compiles; the POOL caps resident conversations. Deployed c769ff2, healthy in 558 s, e2e 33/33.

• Measured live, and one prediction corrected. expert cache space 14.87 GiB / 6,496 slots - the arena prediction landed exactly. But GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99x (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted: the per-token KV price is shape-dependent - 7,701 B/token at MAXLEN 262144 (6,100,000,000 B -> 792,069 tokens) vs 7,892 at 200704, ~2.5 % dearer. Never extrapolate a concurrency figure across geometries.

• Smoke-verified on the live endpoint: three concurrent 67-token requests answered in parallel (~4.1 s each) - the 3 slots are real; native tool_calls (get_weather {"city":"Seattle"}); thinking in reasoning with clean content; reasoning_effort accepts only {xhigh, medium, low} and answers 400 for anything else (re-measured; the fork's message names the set).

• Harness contract followed the window: models.yml contextWindow 262144 -> 200704 (compaction trigger 140,492 + 36,864 = 177,356 <= 200,704). Side effect: frank-archon's compaction request now fits inside a frank-zealot slot again, which it had stopped doing when frank-zealot went to 3 x 180,224.

• Comment-only follow-up 295d578 recorded the measured numbers and was deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.

• Rollback for the pre-shape-D fp8+MTP4 era stays git tag frank-archon-fp8-mtp-20261001. Runbook dankserver-qwen38-fp8-expert-offload-tuning.

2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service

Found by accident, and it had been latent since the secure-connections posture was set: deploy_plex gated on curl -fsS http://127.0.0.1:32400/identity, which cannot succeed. PMS drops plain HTTP off-loopback - the probe's connection arrives on the container's eth0 - so Plex accepts the TCP connection and then answers with nothing, and curl reports "Empty reply from server" (52).

• Trigger: a bare ./scripts/deploy.sh dankserver (no --service) drains the whole pending backlog, not just the service you were thinking about. plex's per-service revision had been behind for long enough that this gate had never run against the current posture. It burned its full 600 s and failed the run; every service after plex in the order was left undeployed.
• Plex was healthy the entire time. https://127.0.0.1:32400/identity returned 200 with the MediaContainer XML in 44 ms while the gate was still timing out. The in-container http://127.0.0.1:32400/identity also answers 200 - loopback is the one path PMS serves in the clear, which is exactly why the host-side probe could not see the difference.
• Evidence chain that settled it: podman exec plex curl http://127.0.0.1:32400 = 200; curl http://127.0.0.1:32400 from the host = empty reply; /proc/net/tcp inside the container showed 00000000:7E90 LISTEN plus stuck ESTABLISHED pairs on 1703590A:7E90 (10.89.3.23, the container's own eth0 address) that never produced a response.
• The repo already knew this law, in a different file: tests/caddyfile.bats pins the plex vhosts to a TLS upstream with the comment "PMS drops plain HTTP off-loopback". The deployer's gate never got the same treatment. Fixed to curl -fsSk --max-time 3 https://127.0.0.1:32400/identity (-k: PMS serves its own self-signed cert on 32400).
• Re-deployed 39f0e0d: plex recorded deployed, the *arr services recorded (compose-recorded on this host, no restart attempted), tautulli deployed, e2e 33/33. Verified after: plex-https=200, plex container up, tautulli healthy, frank-archon untouched (health=200, NRestarts=0).
• Operating lesson: deploy.sh <target> with no --service is a fleet-wide drain. It restarts every service whose recorded revision is behind HEAD and whose paths changed since. To touch one service, name it.

2026-10-02 — frank-archon shape E: --expert-precise-memory (+1,026 arena slots), a mounted compile cache, and the P2P answer

Shape D left one question open — the plan's overhead term was reserving 2.35 GiB of compile headroom the boot never used — and the answer to Frank's free-VRAM question arrived together with a boot-cost bug that had been costing 181 s on every restart since the cutover.

• --expert-precise-memory is real and it fits (option A). Boot-probed on the shape-D base with production stopped, the candidate booted AS the service (FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh, trap-guaranteed restore), the numbers read off the engine's own plan lines:

  plan line	shape D	shape E
  runtime overhead (non-torch + buffers + activation)	7.60 GiB	5.25 GiB (precise (gfx1201 TP2 nbt4096/seqs16/len262144))
  expert cache space	14.87 GiB	17.22 GiB
  slots	6,496	7,522 (+1,026, +16 %)
  swap region	—	13.7 GiB / 5,986 slots, 133-134 per layer
  boot	—	healthy in 420 s, no OOM

  The flag also prints 'expert_precise_memory': True in non-default args, so a silent argparse drop cannot hide. It was refused on 2026-10-01 only because the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the flag was the last stack over the line, not the cause.

• The pool was NOT cut (option A) because the pool is not an arena lever. A 2.2e9 arm was probed in the same window and reports 277,686 tokens, 1.38x — measured, rejected as the shipped value, kept as the conservative fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and gains the arena, which is the shape D follow-up had actually been looking for.

• THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each. Measured: a fresh container has no /root/.cache at all (No such file or directory); after our boot it holds /root/.cache/vllm at 1,154,714,333 B in 4,665 files plus /root/.cache/comgr at 229 MB, inside the ephemeral layer, and podman rm -f (every deploy, restart, reboot) discards them. The engine names the loss itself: Compiling a graph for compile range (1, 1024) takes 173.19 s ... torch.compile took 181.20 s in total ... init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s). The serve script now mounts FRANK_ARCHON_CACHE_DIR at /root/.cache — the parent, so the vLLM AOT/torch_compile tree (VLLM_CACHE_ROOT, default /root/.cache/vllm) and the comgr JIT cache both persist, and nothing is hidden because the image has no such directory. Same class of fix as the recipe's /cache/{triton,vllm,inductor} volumes; the compile RANGE is part of the cache key, so an MNBT change costs one recompilation, not a permanent tax.

• GPU_MAX_HW_QUEUES=2 (HSA hardware queues per GPU) added at Frank's direction. Verified a real knob rather than a no-op — the literal is read from /opt/rocm/core-10.0/lib/libhsa-runtime64.so — and shipped deliberately alongside the 32x64 gather geometry. Not A/B measured.

• P2P: the hardware supports it, and the engine is already using it. The live log shows r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…) on BOTH ranks and Using ['R4D_AR', 'PYNCCL'] all-reduce backends — R4D's P2P path, with RCCL as fallback, measured at 45.274 GB/s bidirectional. Root ports 00:01.1/00:01.3 both report ACS ReqRedir+, so P2P hairpins at the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board swap. Kernel support confirmed: CONFIG_HSA_AMD_P2P=y, CONFIG_PCI_P2PDMA=y, CONFIG_HSA_AMD=y, CONFIG_ZONE_DEVICE=y, iommu=pt on the cmdline.

• Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:

  • Acted on: persistent compile caches (above — the one real operational gap), and GPU_MAX_HW_QUEUES.
  • Retracted: pinned --compilation-config cudagraph_capture_sizes was on my list; measured, it is not a gap here — capture is 3 PIECEWISE + 2 FULL graphs, Graph capturing finished in 3 secs, took 0.31 GiB (0.11 GiB on the lighter rank). No 15-52-size ladder to cut.
  • Not acted on — measured candidates, none shipped (no bench window; these are CPU-side and our decode is gather-bound, so a spin-wait stealing cores from the gather threads is plausible but unproven): OMP_NUM_THREADS=8 (defaults to all 16 logical CPUs on this 7700X), HSA_ENABLE_INTERRUPT=1, HSA_ENABLE_MWAITX=1, HSA_ENABLE_SDMA, HSA_NO_SCRATCH_RECLAIM, HSA_ENABLE_IPC_MODE_LEGACY. All six HSA names verified real — the literals are read out of /opt/rocm/core-10.0/lib/libhsa-runtime64.so — so they are knobs, not no-ops; any of them needs its own A/B before shipping.
  • Deliberate divergence, keep ours: no MTP (the recipe has it; measured fastest here), qwen3_xml vs qwen3_coder (ours verified live), no --override-generation-config (server-side sampling defaults), published port + Caddy rather than network_mode: host.
  • Non-gaps: shm_size 32gb (ours is --ipc=host), memlock -1 (same), SYS_PTRACE (same), /dev/kfd + /dev/dri (same), --enable-chunked-prefill (V1 default), plecache volume (the flag is not even passed), and ROCR_VISIBLE_DEVICES=1,2,3 vs our HIP_VISIBLE_DEVICES=0,1 — verified equivalent here: rocm-smi --showid puts the two R9700s at GPU[0]/GPU[1] and the 7700X iGPU at GPU[2], so our indices are the right two cards.

• Harness note, because it cost two probe attempts: a probe that reported an instantaneous candidate death was a logging artifact of the inline SSH script, not a crash — the container and its serve script must be polled by recorded id and their logs dumped on the exit path, or a failure looks like a bad flag. The flag was fine both times.

• Live: health=200, NRestarts=0, arena 17.22 GiB / 7,522 slots, KV 398,658 tokens / 1.99x. Docs: dankserver-qwen38-fp8-expert-offload-tuning.

• Deployed 1ac4d48, e2e 33/33, and the live plan matched the probe exactly: 'expert_precise_memory': True, runtime overhead … 5.25 GiB, expert cache space 17.22 GiB / 7522 slots, swap region 13.7 GiB / 5986 slots, GPU KV cache size: 398,658 tokens … 1.99x, pool 3,150,000,000, MNBT 1024. The precise overhead and the arena growth are now production numbers, not a probe's projection.

• The compile-cache fix was then proven on the same endpoint, two consecutive boots of the deployed shape: cold Compiling a graph for compile range (1, 1024) takes 179.24 s → torch.compile took 188.26 s; warm Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s → torch.compile took 3.97 s in total → init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s). The host dir held 1.4 GB in 4,707 files after the cold boot. Time to /health is dominated by the 172.8 GiB payload load (the warm boot still needed 490 s), so the honest claim is ~184 s of inductor/AOT build recovered per restart — not a proportional boot cut.

2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000

• Ask: two concurrent 200,000-token conversations on exact attention state. Sized from the live fp8 boot ledger, not arithmetic: the paged groups cost 7.3125 KiB/token per rank (kv_attn 26.00 KiB per 4-token block + kv_qsa_bkey 3.25), so the 3,543 MiB pool bought 386,904 tokens = 1.93 sessions of the served window.
• Engine door: --kv-cache-dtype takes ONLY bf16|fp8 — one dtype for the whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot runs) has no radiance equivalent. Measured bf16 rate is 13.8125 KiB/token = 1.89× fp8, not 2×: the QSA block key (13 layers, 3.25 KiB per 4 tokens) does not follow the dtype. Pool formula tokens × rate + 110 MiB × seqs (recurrent state) + 448 MiB (prefix checkpoints) reproduces both boots within 0.1 %, and is now a test seam.
• Deployed (0bf26b0 + docs): pool 3543 → 6912 MiB (462,752 tokens; ledger "backs 2 sequences all at 200000 tokens at once"), weights slab 26071 → 22208, host pool 40960 → 44928 (the evicted experts land there; pinned pool slots 7,190 → 10,375), prefix host 4096 → 8192 (the entry grows 58.50 → 110.50 KiB), TTM floor 140000 → 150000 (pinned ledger ~147 GiB, cap 204,800). weights + kv pinned to the measured ceiling 29,120 MiB/card; cards settled 1.85 GiB free — more than the fp8 boot's 1.39, because the total claim dropped. Half the cost is refunded by the engine: idle KV VRAM is the expert cache's spillover, and the bigger pool lends 6.08 GiB where the fp8 pool lent 2.69.
• --max-num-seqs 3 → 2: the fleet's "seqs 2 cannot boot" law is the retired tcclaviger vLLM fork's torch.compile query_start_loc constraint, not radiance's — radiance booted seqs 2 in the fp8-native era and the 2026-10-06 sweep had 2 and 3 tied inside ±3 %. Bonus: the fused all-reduce is declared over 2 × (1+2) = 6 rows, inside the 25-row bound where the wht6 wire is bit-exact.
• Verified live (no scratch container, no bench): /health 200, served id frank-archon; 2 × 196,919 concurrent conversations each answered the marker buried at the head of its own prompt (393,837 tokens resident, above the fp8 pool's 386,904); decode at 51.5k depth 94.1 cold / 102.3 warm / 117.6 prefix-hit t/s, draft acceptance 0.7284 / 0.7525.
• fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in (the frank-zealot ubatch method, argv + ledger page size verified): prefill −29 % (4,379 → 3,126 cold, 4,864 → 3,333 warm), decode inside the noise band on two of three passes, TTFT at the full window +25 % (the pair: 110.5 s → 138.1 s), draft acceptance up 0.661 → 0.740. The price is prefill — a 14.8 % smaller resident slab, and prefill is the pass that touches every expert — the gain is exact state plus acceptance. Open call for Frank: keep bf16, or take the cheap fp8 2 × 200k fit (pool 3,648 MiB, ≈ free) and leave KV quantized.
• Harness law re-learned: radiance puts the thinking block in reasoning_content, so a probe that reads content on a small budget sees empty text with finish_reason: stop — a healthy server scored 0/2 until the probe set chat_template_kwargs.enable_thinking=false. Tool committed as scripts/probe-frank-archon-kv.py.
• Tests 368/368, e2e 33/33 (one unrelated homepage retry). Docs: frank-archon-radiance-swift15-w4dn8, and the LLM index entry corrected (the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).

2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only)

• Ask: replace the served model with StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe — the engine author's published rad-convert of the OFFICIAL Qwen3.8-Flash-Next (w4nl codebook experts, ten bf16-protected/layer, int8 trunk, PLE/MTP/vision in-container, card KL 0.0789 / top-1 92.85 %) — no other argument changes, mindful the operator session runs on the endpoint being swapped.
• Disk: Frank authorized one rm — the spent Swift1.5 BF16 source (336 GiB) and the two-era-old qwen4exp-fp8-native.rad (173 GiB) left /home/dank/models; 150 GB → 658 GB free. fp8-native is now rebuild-only (fetch + build scripts); rollback for THIS swap is the w4dn8 .rad (on disk) by reverting the commit.
• Fetch: new scripts/download-frank-archon-iq4r.sh (hfget, exact bytes 121,969,901,568 + Hub LFS sha256, --verify-only default) as transient unit iq4r-fetch; verified, staged sidecar files kept as provenance.
• Shape carried VERBATIM (25600/5120 pair, 44928 pool, fp8 KV 3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5). Tests re-keyed (frank_archon_unit.bats pins the new payload id + sha256, keeps the w4dn8 rollback provenance), homepage card + trio lore + both indexes updated; new runbook frank-archon-radiance-qwen38-iq4r. Commits 6dcf4a5 (swap) + f44c398 + c5ac072 (docs), full bats green.
• Cutover self-severed by design (detached deploy.sh survived the session death when frank-archon restarted); engine healthy in ~100 s. First-boot ledger: w4nl loads on pin 1.0.13 (no pin bump); kv_attn 26.00 KiB fp8 @ 607,740 tokens (identical geometry); slab 17,610/17,658 slots per rank (> w4dn8's 17,548 — fewer bf16-protected experts pack more 4-bit table); sampler 9 rows; MTP 2/step; prefix tier RECOVERED 900,176 blocks across the swap (arch+tokenizer unchanged). Vision probe green (reasoning read the solid-red PNG; the 64-token budget went all to thinking — fleet law).
• e2e gate went red on three dankclaw specs — NOT the swap: dankclaw rebooted 01:41 and the gluetun boot-race dependency-failed searxng, marreta, and marreta-browser (jobs consumed while gluetun's start job retried; Restart=always never armed because the services never ran). Cascade restarted; skill codified (gluetun-dependent-boot-race-recovery). Gate rerunning against the repaired fleet; frank-archon's own health gate passed every deploy. Open: decode/prefill ladders vs the w4dn8 record, probe leg D — scheduled probe window.

2026-10-15 — authentik: two more human accounts (Dankclaw)

• Live — mitch (pk 22) and eric (pk 23) provisioned via ak shell ORM, no repo objects. Each <user>@franksdank.cloud in the human-user template shape (is_active=True, display name, password via set_password on stdin only — never argv, so nothing lands in ps), groups Non-Admin Users + jellyfin_users + openwebui_users; no admin group granted. Verified through the exact PasswordStage code path (authenticate() accepted each password and rejected a wrong one), app_entitlements(open-webui) → Open WebUI Users for both (identical to the ridley precedent), and is_superuser=False / path=users. Zero proxysession rows for either — groups were set before any login, so no claim purge was due. Downstream: Jellyfin visibility rides the LDAP outpost cache (expect lag); OpenWebUI mints their accounts as role pending on first login.
• /serverAppConfigs-docs/Infrastructure/Authentik.md — human-user template census extended with pk 22–23.

2026-10-17 — frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)

• Ask: evaluate prism-ml/Ternary-Bonsai-2-27B-gguf (PQ2_0) as frank-zealot's payload, and specifically measure 2 × 200,704 with q8_0 K + q8_0 V on the RTX 3090. Fetch script scripts/download-frank-zealot-bonsai2.sh (exact bytes + sha256, --verify-only) landed first; payload verified at /home/dank/models/Bonsai2-PQ2_0/ (7,206,168,928 B + 629,246,976 B mmproj).
• Runtime door: the file is file_type 141 with 402 tensors of custom GGML type 142 plus prism.hadamard.* metadata — stock llama.cpp refuses it, so the PrismML vendor build 10754 (2459f68b5) is forced, not chosen. A source build with GGML_CUDA_FA_ALL_QUANTS=ON was started and then cancelled as unnecessary: the vendor build already serves q8_0 V under flash attention at the exact 34,816 B/token (ledger-confirmed).
• Trial-window law cost one boot: the scratch env invented KV_CACHE_TYPE where the serve script reads CACHE_TYPE_K/CACHE_TYPE_V, so the script's :=f16 default silently won and every arm booted f16. That failed window is now the measured f16 wall: cudaMalloc OOM on a 25,088.00 MiB KV request, engine fit 33,198 vs 23,558 free.
• Four-arm census (production stopped, trap-guarded restore, bg_12): 2 × 200,704 q8_0 = 22,664 used / 1,461 free (DEPLOYED); 1 × 262,144 q8_0 = 18,130 / 5,995 (fits, but serializes compaction); 2 × 262,144 q8_0 ≈ 25,758 projected (no fit); 2 × 200,704 q4_0 = 16,392 / 7,733 (fallback rung). Recurrent DeltaNet state bills ~150 MiB per SLOT (context 13,627 − KV 13,328 for two slots), which pool arithmetic alone misses.
• Behavior probes green: exact-instruction chat, reasoning_content present, tool_calls with valid JSON arguments (no fork KNOWN_ISSUES loop/500), 512 px solid-red PNG answered "Red", needle retrieval exact through a 25,960-token prompt, 166,027-token prefill clean in 240.0 s. Rates: 76.6 t/s single-stream (9B ran 94.4), 1,258 t/s at 35k, 691.7 t/s at 166k, 109.4 t/s aggregate at two concurrent streams.
• New fleet laws recorded (AGENTS.md + runbook): the mmproj's 873.10 MiB worst-case is NOT inside the engine's fit projection; the first request after a boot is a warmup artifact (2.6 / 37.7 t/s measured); a 64-token thinking budget makes a thinking model look blind; the Bonsai-2 template reads reasoning_effort (xhigh/medium/low, raises otherwise) so the harness fold became load-bearing on this endpoint, and it reads preserve_thinking.
• Cutover: scripts/frank-zealot.env (payload + CACHE_TYPE_*=q8_0 + ledger lore), scripts/deploy-remote.sh (runtime/model/mmproj gates re-keyed, fork hint), tests/frank_zealot_unit.bats (payload pair pinned; KV assertions now env-derived instead of an f16 literal), harness models.yml (extraBody 1.0 /0.95/20 + preserve_thinking, effort-map comment corrected — contextWindow 200704 and maxTokens 16384 unchanged), new runbook frank-zealot-llama-prism-bonsai2-pq20, indexes re-keyed, 9B era retired in place. Full bats green (371).
• Base alignment: Frank chose the OFFICIAL censored payload — publisher provenance over refusal-freedom. The 9B it replaces was uncensored, so the homepage card and its warning contract were re-keyed with it: zealot's card now carries the base-alignment notice, the shared legal warning belongs to dragoon alone, and the homepage_config.bats equality (zealot.warn == dragoon.warn) retired with the payload it described. The CRACK re-export (byte-identical geometry per header census) stays on disk as the one-line MODEL_FILE fallback if refusals bite.
• Deployed 19:44 (--service frank-zealot --service homepage): production boot reproduced the census exactly — 22,664 used / 1,461 free, fit projected 21,439 leaving 2,119 ≥ 1,872, KV 13,328.00 MiB at n_ctx_seq = 200,704, n_slots = 2, projector loaded, alias frank-zealot on /v1/models. Live probes: exact pong + reasoning_content, tool_calls {"city":"Seattle"}, 512 px PNG → Red, 75.8 t/s decode (1 % off the trial arm). e2e 33/33.
• Two live-path stragglers the cutover also had to catch: serve-frank-zealot.sh still DEFAULTED to the 9B paths and f16 KV (a defaulted run would have served the retired payload — now re-keyed to the shipped trio), and the OpenWebUI model row's meta.description is live DB state naming the 9B (re-keyed, backup webui.db.bak-20261017a; no open-webui restart forced — the cached list picks it up on the next one, and forcing it logs every family session out for one cosmetic string).

2026-10-17 — dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)

• Ask: "dump the swift rollback and the GLM GGUF — make sure the model caches and agenttrash are empty as well." Preflight first: lsof empty on both payloads, serving trio names only the iq4r path, and a podman layer census proved no deleted path backs an image container (the 121 GB 16b6663d… layer is the ENGINE image, not a model blob).
• Staged to /home/dank/.AGENTTRASH/20261017-dump-payloads-caches, then purged with Frank's approval: the Swift1.5 w4dn8 .rad (113.6 GiB, rollback target), GLM-4.7-Flash-UD-Q5_K_XL.gguf (21 GiB, /opt/vllm dir removed — last non-iq4r model on the box), radiance-state kvcache tiers (69 GiB + 61 MiB, derived), ~/.radiance-cache-w4a8-* (11 GiB stale-pin compile cache), ~/.cache/frank-primary (4.7 GiB stale-name cache). Also emptied every .AGENTTRASH on the host (14 dirs: dated batches, quadlet/config holds, both checkouts, /etc/caddy). 543 → 763 GiB free.
• Recovery: the w4dn8 payload is byte-exact on the Hub as frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 (122,071,973,888 B, LFS sha256 09dd9306…) — rollback is now a re-download, not a local file. Corpora (grams 41 G, kld-ref 24 G, calibration JSONL) deliberately KEPT: re-convert and KLD-gate inputs.
• Post-checks: frank-archon active, /health ok; iq4r serving payload byte-size intact; live kvcache tier (122 MiB) untouched; no process references a deleted path.
• Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook index rows; iq4r runbook Rollback rewritten (HF re-fetch procedure — the fetch script's REPO/RAD_NAME/STAGE/DEST/SIZE/SHA256 constants are hardcoded, so retargeting at the mirror is a script edit); w4dn8 runbook header corrected (was still "LIVE", never updated at retirement) and its Rollback table rebuilt.

2026-10-17 — frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool

• Ask: move frank-dragoon to Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated with the card's recommended settings, save memory, keep the n-gram table disk-backed. Census first: the endpoint carries only the slow + security-reviewer roles (the interactive workhorse is frank-archon), the table was ALREADY disk-backed (mapped 51.2 GB, NOT pinned (Pin::None)), and HALOGEN_FLASH_PIN_TRUNK=0 is rejected by the v2 loader — so the only real lever was the KV pool: 524,288 → the card's 262,144.
• Filename trap caught before the fetch: this repo name carried two different artifacts. The writer-transplant era's file measured 66,687,678,464 B; the native 2026-10-11 conversion is 66,687,678,432 B — 32 bytes apart. Byte-exact gate + LFS sha256 b5bd925f… in scripts/download-frank-dragoon-v2abl.sh, which also HARDLINKS the peonist n-gram/vision/tokenizer trio (the release publishes only the main file; a hardlink is a regular file, so the no-symlink serve gate passes without a 47.7 GiB copy).
• Companion risk adjudicated by the engine, not by argument. The card does NOT establish the stock peonist companions as matched to a native conversion and publishes no source-matched pair, so the A/B ran the checkpoint tools on a stopped service under one knob set (scripts/census-frank-dragoon.sh): verify OK, NIAH 18/18 vs 17/18, PPL 3.968 vs 3.877 (+2.3 % — the finetune's edge is gone), KL vs the swift15 logits mean 0.155 / median 0.057 with top-1 85.1 % = two sound models disagreeing on content, not a corrupt forward pass. Behavior battery (scripts/battery-frank-dragoon.py) on both boots: refusals 0/5 both, tools/vision identical, needles hit at both depths, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 (+12.6 %), thinking tokens 1,476 vs 784 (the finetune compressed reasoning ×1.9).
• Boot window run through a transient FRANK_DRAGOON_ENV_FILE drop-in — never a scratch port (one iGPU, one server) — with the stop+mask law and a trap-guarded restore (scripts/trial-frank-dragoon.sh). Masking fails on a relinked unit file, as documented; stop alone was enough.
• Deployed at 22:20 (fcd7630, --service frank-dragoon): healthy in ~12 s, e2e 33/33. Production ledger: 62.9 GiB weights locked (0.8 vision) + 7.2 GiB KV (ONE 262,144-position pool, 6,656 MiB, two slots) + 5.7 GiB working = 75.8 GiB in all, no clamp line, n-gram 47.68 GiB read from disk, prompt cache ON with 12 entries, MTP depth-2. The trial drop-in was moved to ~/.AGENTTRASH/20261017-v2abl-trial/ before the deploy so the repo env governs (DropInPaths= empty). Live probes: 27*59 → 1593, tool call {"city": "Seattle"}, 512 px red PNG → Red (320 prompt tokens).
• Files moved: env (payload dir/checkpoint/bytes/sha256/pool + the measured A/B in the header), serve script + deployer (paths, fetch hints), unit Description, seam (frank_dragoon_unit.bats: pool law relaxed from pool == slots × ctx to ctx <= pool <= slots × ctx, fetch-script name), new runbook frank-dragoon-flash-server-v2-native, swift15 page re-banned RETIRED, both indexes + AGENTS.md re-keyed, homepage card, models.yml comment (numbers unchanged: window 217,088 / cap 45,056 — the per-REQUEST ceiling did not move). OpenWebUI frank-dragoon row description re-keyed (backup webui.db.bak-20261017b, no restart forced).
• Two stale claims fixed on the way: the retired-era rows said 200,192 ctx (that ceiling was raised to 262,144 on 2026-10-07, inside the same era), and the root index still called the halogen-27B page ACTIVE.
• Cost of the swap, stated plainly: two MAXIMUM-length conversations no longer share the endpoint (one 262,144-position pool behind both slots), PPL +2.3 %, thinking ×1.9. Bought: 7.4 GiB of RAM back as page cache for the table long prompts read, +12.6 % cold prefill, retrieval 18/18, the base checkpoint itself instead of the finetune.

2026-10-17 — frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix

• Ask: lower frank-dragoon's context "to 196k". Settled on 200,704 for BOTH HALOGEN_CTX and HALOGEN_KV_POOL_POSITIONS (49 × 4096 — the per-slot rung frank-zealot runs; the card recommends pool == ctx, and that equality is where the memory is). Nothing in the fleet needs the bigger window, and the v2 checkpoint serves anything up to its native 262,144.
• Boot ledger (7c4dbeb, status=deployed): reserving the KV pool, 200704 positions (about 5.5 GiB), 62.9 GiB of weights … 5.5 of KV pool, 5.7 of working memory, 74.1 GiB in all, prompt cache entries 111 MiB at the new ctx, no clamp line. My arithmetic predicted 5.4/74 — the ledger is quoted instead.
• Harness moved WITH the ceiling (400-zone law): models.yml contextWindow: 155648 = ceiling − 45,056; compaction now fires at 85 % = 132,301 est ≈ 136k measured, and the 180k compaction target of the 262,144 era is unreachable because the window itself is smaller. maxTokens 45,056 and the coupled cap/timeout 65536/7200 untouched. Worst-case harness request = 1.03 × est + min(45056, 151552 − est) ≤ 156.2k tokens vs the 200,704 ceiling.
• Probes on the live endpoint: over the ceiling → 400 … it is 297109 tokens and the context is 200704, 96405 tokens over in 0.86 s (preflight rejects before any prefill, so probing costs the endpoint nothing); harness-shaped admission → 122,679-token prompt with max_tokens: 45056 (reservation 167,735) accepted, 81.1 s wall (≈1,513 t/s cold prefill), 27*59 answered 1593 from inside a 122k-token haystack, finish_reason: stop.
• Seam re-keyed (frank_dragoon_unit.bats): the fit-law test now proves the invariant at the new pair — ctx ≤ pool ≤ slots × ctx and harness window ≤ ctx − 45056 — so a future raise of one without the other fails the gate. bats 371/371.
• Deploy verdict: the endpoint dispatched healthy (status=deployed, /health 200), but the [3/3] Playwright gate failed on 12-crawl and the deployer correctly said DEPLOY INCOMPLETE. Root cause was NOT the endpoint — it was Plex Web, signed out in the qa-e2e profile, running its own sign-in flow, which broke the spec two different ways:
  1. it redirects the top frame, so Chromium cancelled the crawl's in-loop page.goto("/") with net::ERR_ABORTED — and that reset sat outside the try, so one host aborted the whole crawl (once as a 420 s timeout, once at 5.4 s); fixed with backHome(), which re-issues on that one error class.
  2. the same hand-off leaves the browser on app.plex.tv (Plex's hosted surface for a session-less client), so the hostname == target law could never settle — this is what the first full-suite re-run surfaced as 1 flaky; fixed with an explicit HANDOFFS landing list. Plex itself was healthy throughout (container up 2 days; 32400 answers HTTPS 401 / plain-HTTP empty reply, which is what Plex Web authenticates against), and the guard contract is still asserted for every vhost: challenge cleared, no authentik uidField, status < 500. Both symptoms recorded as gotcha 6.
• Gate after both fixes: 33 passed (1.2 m), crawl green on the first attempt (6.5 s) — the earlier DEPLOY INCOMPLETE is cleared, and the deploy marker stays at 7c4dbeb because every later commit is docs/tests only, which the planner treats as a no-op for the service.
• Cost of the cut: two maximum-length conversations never coexist (one 200,704-position pool behind both slots; two of ≤100k still do). Bought: 1.7 GiB back as page cache for the 47.68 GiB n-gram table — 9.1 GiB against the 2026-10-06 shape's 524,288 pool — and a smaller prompt-cache entry.

2026-10-18 — issue #3 research: AMD PerfOpt retires the reason for amd_iommu=off

• Ask: does a recent kernel change remove the IOMMU translation penalty that justifies the 2026-09-11 amd_iommu=off boot line, i.e. can the XDNA2 NPU come back? Answered and documented; no host change — the fix is not installable from the Ubuntu 26.04 archive.
• The change is AMD PerfOpt (spec §3.4.9, MMIO 0x016C bit 13, gated by extended-feature bit 45): merged via the iommu tree 2026-09-24, first release Linux 7.4. It stops GPA→SPA translation for the privileged iGPU only and leaves the IOMMU fully armed for every other device, NPU included. The only consumer is amdgpu_device_use_perfopt(), which requires AMD_IS_APU and the identity domain — so PerfOpt rides on top of iommu=pt, it does not replace it. A 2026-09-28 follow-up deletes the amdgpu.iommu_perfopt knob and makes it compulsory.
• Local census (read-only): kernel 7.0.0-38-generic; zero perfopt symbols in its System.map; linux-image-generic and linux-generic-hwe-26.04-edge both pin 7.0.0-38.38 → 7.4 needs a mainline PPA or a local build. CONFIG_IOMMU_ DEFAULT_DMA_LAZY=y (STRICT unset). /dev/accel/accel0 already exists as crw-rw---- root:render 261,0 with dank in render, so the quadlet law is AddDevice=/dev/accel/accel0 + GroupAdd=keep-groups — the ComfyUI /dev/kfd pattern, no new udev rule.
• Three premises of the issue corrected. (1) The 13–16 % is the vendor's A/B against their previous boot line, which our record never identified — we copied the figure on 2026-09-11 without ever measuring here. Strict flush and bounce buffering are not the mechanism either (Ubuntu ships lazy DMA, and a coherent iGPU on unified memory takes no SWIOTLB bounce). What identity mode actually costs the iGPU is the 1:1 page walk plus ATS/PRI/PASID/GCR3 being switched on at attach — the same four the PerfOpt enable path switches back off (the skip_caps label in attach_device), and why it arms only "when an integrated GPU is already in identity mode". (2) The NPU does not have to wait for 7.4: iommu=pt enables the IOMMU, which is all err=-19 asks for. Making passthrough the AMD default was rejected on the list (Robin Murphy, 2026-03-26), so pt stays the sanctioned route. (3) The 460 tok/s baseline is stale — the 2026-10-17 native-v2 payload runs 1,513–1,604 t/s cold prefill — so the window measures off vs pt locally and inherits no number.
• Silicon check: our IOMMU is 1022:1508 Strix/Strix Halo rev 02, and Phoronix's Strix Halo desktop arm measured only 2–4 % from PerfOpt (18–23 % on Strix Point / Krackan laptops) — the modest arm is the one that describes this box. Kernel pick: skip 6.18–6.18.7, where an IOMMU regression disables SVA outright.
• NPU legality under identity mode: amdxdna binds via iommu_sva_bind_device()
  • PASID + GCR3 into its own IOPT_TYPE_SVA domain, independent of the default domain type (drivers/accel/amdxdna/aie2_pci.c); err=-19 needs only an enabled IOMMU, which pt gives.
• Self-correction inside the same session: the first pass — and the first version of the posted comment — claimed iommu=pt alone already avoids the cost. It does not. Patch v2 1/2 shows an identity attach keeps the page walk and enables ATS/PRI/PASID/GCR3, so pt recovers most of the gap and PerfOpt is the last part of it. The comment was edited to the corrected text.
• Artifacts: new section in serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md (+6 citations, front-matter re-keyed to "vendor-claimed, never measured locally"), frank-npu.md § Why it is retired corrected — the "pt is not a middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, a 4-step maintenance-window procedure that measures locally instead of inheriting the vendor number, and the plan posted to Gitea issue #3 (comment 23, then edited). bats 371/371; commits 7643381 plus this correction. Nothing deployed — no service was touched.

2026-10-18 — boot line flipped: Dankclaw now boots iommu=pt

• Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the research demanded. Provenance first: /etc/default/grub had never been in the repo while the IOMMU line was the most consequential kernel decision on the box, so it is now tracked at dankclaw/host/grub (fetched byte-exact, 1581 B) and moves only through scripts/set-dankclaw-bootline.sh — diff, install 0644 root:root, update-grub, verify the default grub.cfg entry, reboot on --reboot only. The deployer plans nothing under dankclaw/host/: kernel state is not a container restart.
• Measured arms, byte-identical traffic. OFF arm at uptime 2 d 23 h: cold prefill 1,585.1 t/s (59,024 tok), warm 208,531, cache ratio 131.6×, needle hit/hit @21,568, decode 37.3 t/s. PT arm after the reboot: 1,616.0 t/s (+2.0 %), warm 214,131, ratio 132.5×, needle hit/hit, decode 33.7 t/s. The corpus was rebuilt after the reboot deleted /tmp (tmpfs) and is provably the same input: 2,891 lines, 1,287,656 B, and the engine billed identical token counts. Verdict: identity DMA costs nothing on this host — the 13–16 % that justified amd_iommu=off for five weeks never existed here. Decode's −9.7 % is instrument noise: three repeats of the same request on the same boot gave 31.8, 46.9, 47.8 t/s.
• Gates after boot: 0 failed units, systemd running, 32 IOMMU groups (31 identity
  • 1 DMA = the Genesys Logic SD reader c2:00.0, a kernel quirk), iGPU c5:00.0 identity group 20 with ATS, interrupt remapping on. frank-dragoon's ledger is byte-for-byte the same shape (62.9 + 5.5 + 5.7 = 74.1 GiB, 2 slots over ONE 200,704 pool, MTP depth 2, prompt cache ON, no clamp; listening at 39.8 s), and Gitea / Caddy edge / Grafana answered 200.
• The XDNA2 NPU is back: amdxdna bound 0000:c6:00.1, firmware amdnpu/17f0_11/npu_7.sbin, amdxdna_accel_driver 0.7.0 … minor 0, group 26 identity, PASID width 10. /dev/accel/accel0 crw-rw---- root:render 261,0 with dank in render. No endpoint deployed — revival needs a quadlet with AddDevice=/dev/accel/accel0 + GroupAdd=keep-groups.
• Deliberately NOT done: the pp_dpm_fclk level-7 pin (it changes iGPU timing for every workload and nothing co-runs on the NPU yet) and any PerfOpt work (kernel 7.0.0-38 has zero perfopt symbols; the 26.04 archive caps at 7.0.0-38.38).
• New tool: scripts/stage-frank-dragoon-eval.sh rebuilds /tmp/halogen-eval byte-identically (corpus + probe PNG + battery) — run it through bash -lc, uv is in ~/.local/bin. Docs re-keyed: decision record (banner, front matter, verification/revert, new § 2026-10-18 execution), frank-npu.md (precondition satisfied, endpoint still absent), AGENTS.md key lesson, both index pages. bats 371/371. frank-dragoon and frank-zealot were live the whole window; only the reboot interrupted them (~40 s to listen, health 200).

2026-10-18 — frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144)

• Swapped prism-ml/Ternary-Bonsai-2-27B-gguf → Ternary-Bonsai-2-9B-gguf at PQ2_0 (Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf, 2,675,410,720 B, Hub-LFS sha256 c79fcdfaf6b7… verified at fetch) plus the publisher's Q8_0 mmproj (624,229,664 B, e0cb6f69…), into /home/dank/models/Bonsai2-9B-MTP/; the runtime moved to vendor fork build 10777 (e3de71522) because the MTP arm needs the build that pairs MTP with a quantized V cache.
• The win is the window, not the quant: --ctx-size 786432 --parallel 3 puts every conversation back at the trained 262,144 (the 27B era sold each slot 200,704), and the fleet's widest context makes :7771 the compaction hub — models.yml contextWindow 200704 → 262144. Measured vs the 27B: decode 167.2 vs 76.6 t/s single-stream, aggregate 280.2 vs 109.4 t/s across the (now three) concurrent streams, prefill 2,877 t/s at 106,418 depth.
• MTP armed, then OFF by measurement. The head is in the file, the fork loads it, and the draft context bills its own f16 KV over the WHOLE pool — 3,072 MiB at 786,432 positions, ledger-confirmed. At UBATCH_SIZE=2048 that squeeze halved decode (167.2 → 109.2 t/s). The documented MTP shape is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 → 213.5 t/s; declined, because +17 % decode is not worth spending 6.2 GiB of the 6,249 MiB free. The arm is recorded in the env header, not left to rediscovery.
• Two bugs the trial caught, both fixed in the repo: ${SPEC_PRESENCE_PENALTY:=0.0} was a default-assignment, so the env value always won and the documented "MTP runs without --presence-penalty" arm was unreachable — now ${VAR=0.0}, which lets SPEC_TYPE=draft-mtp drop the flag. And up{job="frank-zealot"} had read 0 for the entire retained Prometheus history: llama.cpp answers 501 on /metrics unless --metrics is passed, so the scrape failed beside a healthy /health and LlmEndpointDown fired for days. METRICS=1 is now the default and the seam pins it.
• Boot ledger: KV 13,056.00 MiB (8 full-attention layers × 4 KV heads × 256 dim, q8_0 K + q8_0 V over 786,432 positions — ledger-exact), settled 17,876 used / 6,249 free; the f16 pair would need 26,112 MiB and cannot boot this card. TimeoutStopSec=180 carries over (SIGTERM ignored past the old default during the 03:38 stop).
• Header census that corrected my own first pass: block_count is 33 (32 trunk + the blk.32 MTP block, which carries a full attention + FFN in q8_0 — that is why the draft context bills real KV), 202 of 442 tensors carry the fork's PQ2_0 type (norms f32, MTP projections q8_0), hadamard block_size 4096 with gdn_grouped=True. The "2.13 bpw / type 142 / 32 layers" figures inherited from the 27B page were wrong for this file and are corrected in AGENTS.md, the env header and the runbooks.
• Docs: new runbook LLM/frank-zealot-llama-prism-bonsai9-mtp.md; the 27B page re-banned RETIRED 2026-10-17 → 10-18 with its bench-window section closed; both index pages, the homepage card, Dankclaw/OpenWebUI.md and the live picker row (DB description + context length 262,144) re-keyed. Rollback: revert the swap commit — the 27B payload and the build-10754 fork tree are both still on disk.
• Deployed 13:16 PDT (deploy.sh dankclaw --service frank-zealot, e2e 33/33) — and the first boot was not the shipped payload: a leftover frank-zealot.service.d/ptq1.conf from the bench window (written 03:29 for the 27B PTQ1_0 arm Frank ordered at 02:36 and measured from 03:14 — see the record below) pointed FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted that arm: 3 slots of 133,888, 21,188 MiB used, batch 3072/ubatch 1536. /health 200, /v1/models echoed the alias, Playwright 33/33 — nothing in the gate reads the payload. Drop-in moved to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/, daemon-reload, restart at 13:19: n_slots = 3, n_ctx_slot = 262144, KV 13,056.00 MiB, settled 17,922/6,203.
• Production measures (engine timings, greedy, ignore_eos): 168.7 t/s decode (168.4/168.7/168.7) and ~300 t/s aggregate across three 512-token streams (102.8 each). My first battery reported 117 and 72 t/s: it timed natural generations that stopped at 24 tokens — wall-clock aggregate on an early EOS reads 4× low. ignore_eos + the engine's own numbers is the only honest way to time this.
• Ledger correction: the shipped shape prints RS 150.75 MiB (3 cells), not the 452.25 MiB my first-pass notes carried — that figure belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot (9 cells). The drafter triples the recurrent state as well as billing 3,072 MiB of f16 KV. Corrected in AGENTS.md (topology row + architecture section), the env header and the runbook, with the production boot ledger replacing the trial's.
• Verified live: chat 27*59 → 1593; where_am_i tool call fired; 512 px solid-red PNG → Red (278 prompt tokens, process_mtmd in the journal); /props n_ctx 262144, presence_penalty 1.0; /metrics 200 with 15 llamacpp:* families and up{job="frank-zealot"} = 1 — the first non-zero sample in that job's retained history.
• OpenWebUI model rows re-keyed in the live DB (backup webui.db.bak-20261018): frank-zealot → 262,144 + the 9B description, and frank-dragoon's context.length corrected 262,144 → 200,704 (it had kept the pre-lowering window from 2026-10-17 while its own description said 200,704). Note the columns are params/meta JSON — the table has no context_length/description columns.

2026-10-18 — frank-zealot PTQ1_0 arm: measured, approved, and never written down

• Correction to the entry above and to my own report. The frank-zealot.service.d/ptq1.conf drop-in was NOT a half-finished experiment. The session archive (01a14c93) carries Frank's own instruction at 02:36: replace frank-zealot with Ternary-Bonsai-2-27B-PTQ1_0.gguf, use the Q8 mmproj, batch/ubatch 3072/1536, "fit as much context as possible in 2 contexts".
• It was the production payload 03:14:39 → 03:38:57 (pid 191358; journal type ptq1_0: 402 tensors, file type = PTQ1_0 — 1.75 bpw ternary, mmproj loaded, 2 × 200,704) and served real traffic. Recovered from that process's own print_timing lines (17 timed tasks): single-stream greedy decode 48.0 and 47.6 t/s at 512 tokens, 47.9 at 229, 49.0 at 451; 25–27 t/s per stream with two streams live; prefill 1,295 t/s @ 30,637 → 943 @ 97,856 → 801 t/s @ 141,365 (one 176 t/s row at 97,024 is a concurrent-slot request, not a clean probe).
• Against the PQ2_0 27B record (76.6 t/s decode, 691.7 t/s prefill at 166k): PTQ1_0 freed the 1,201 MiB its fetch script promised and paid −37 % decode — the publisher's "dense ternary unpacking loses on Ampere" warning, now confirmed on GA102 — while prefilling deep prompts faster than that record ever managed. Frank scratched the whole 27B plan at 03:33 ("scratch the current frank-zealot plan completely… 9B… enable mtp… if you can possibly fit a third context, do it"), which is what shipped at 13:16.
• The durable failure is mine: those numbers lived only in /tmp and the journal, never in the repo, so the next agent on the host could not tell an approved payload from a rogue drop-in and called it a hijack. The runbook deploy ledger now carries the measurement; /home/dank/models/Bonsai2-PTQ1_0/ stays on disk as the 27B rollback and the drop-in stays trashed.

2026-10-18 — frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot

• ./scripts/deploy.sh dankclaw --service frank-dragoon — e2e 33/33. The drop-in check the zealot incident bought ran FIRST: DropInPaths empty, so the repo env governed. Boot ledger: reserving the KV pool, 401408 positions (about 11.0 GiB), 2 slots over ONE 401408-position KV pool, each request up to 200704, and 62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of working memory, 79.6 GiB in all; prompt cache ON (12 entries × 111 MiB at ctx 200704 = 1.3 GiB), n-gram 47.68 GiB still Pin::None, no clamp line.
• Warm reuse survived the raise, measured right after the deploy: one 101,987-token wikitext prompt through /v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on each of two repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so a warm turn prefills nothing and the doubling evicted nothing the cache needed. The RAM price is still real: host MemAvailable fell 26 → 22 GiB, because every pool position is RAM the n-gram table's page cache loses.
• Every document quoted ~10.8 GiB, the ~28 KiB/position arithmetic; the engine reserves 11.0 GiB and its preflight bills 11.3. Re-keyed in AGENTS.md (topology row, KV-pool law, memory ledger), the frank-dragoon.env lore block, both index pages, the iommu page and the runbook, and every "committed but not yet deployed" clause was retired with the measurement.
• The same sweep caught the LLM index still describing frank-zealot as Bonsai-2-27B at 2 × 200,704 in its intro paragraph and topology diagram — the 9B swap of the same day had not reached those two spots. Re-keyed: 8 full-attention + 24 Gated DeltaNet + the blk.32 MTP block, 3 × 262,144, tensor type 141.
• PTQ1_0 provenance corrected (see the dedicated section above): the arm that hijacked the zealot deploy was a repo-tracked, Frank-ordered, measured candidate — scripts/download-frank-zealot-bonsai2-ptq1.sh is committed, and its header now carries the recovered numbers (≈48 t/s decode, 801 t/s prefill at 141k) and the verdict: −37 % decode against PQ2_0 for 1,201 MiB freed, then scratched by Frank's 03:33 order in favour of the 9B. Nothing is owed on it.
• OpenWebUI: both decoration rows re-keyed in the live DB. The values live inside JSON columns — params.context.length and meta.description — not in fields named for them, and there is no sqlite3 binary on the host, so python3 stdlib + Connection.backup() (webui.db.bak-20261018). Zealot 200,704 → 262,144 with the 9B description; dragoon 262,144 → 200,704, which had kept the pre-lowering window while its own description said 200,704.
• Census of the halogen HTTP surface while doing it, because the fleet bench law quoted the wrong engine: frank-dragoon answers /health, /metrics, /v1/models, /v1/chat/completions(POST) and 404s /ping, /props, /completion, /tokenize; the chat response carries no Server-Timing header (server: uvicorn, only date/content-*). So the /completion timings object and the ttft;dur= header are llama.cpp instruments — frank-zealot's, not dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine owns which instrument.

2026-10-18 — frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)

• Voice-over goal, phase 1 (Gitea issue #2). New service frank-tts on Dankclaw: dankclaw/voiceservice/ (stdlib-only app.py + Containerfile + rootless quadlet), scripts/frank-tts.env, tests/frank_tts_unit.bats (12 tests). One OpenAI-shaped POST /v1/audio/speech surface; voice=<name> is honoured only when /voices/<name>/reference.wav exists, otherwise a 400 listing the voices that are — no silent fallback voice. voice_id is the harness alias; response_format=mp3 really is mp3; oversized input is a 414, not a half-narrated sentence; /health reports backend + voice list; no audio ever reaches the log. Two backends behind it (TTS_BACKEND=omni-http forwards to the host's :8880 engine with the voice as base64 reference_audio; llama-tts-cli runs a binary per request at nice -n 19), both exercised through stubs — no GPU, no containers in the seam.
• Naming law applied (Frank's call): treat it as a first-class model endpoint → alias-named frank-tts (TTS endpoint, so not frank-asr), port 7772, joining 7770 frank-dragoon / 7771 frank-zealot; the engine upstream keeps 8880 and is deliberately not a fleet endpoint. The internal knob names stay SHIM_* (they name the façade component, not the service).
• Voice-pack law recorded in the env header: a pack needs reference.wav (24 kHz mono, 18–25 s) and the pinned engine's own speaker artifact — required, beside the WAV, never replacing it — plus manifest.json provenance whose offsets are always past the first 10 s of the source (the source-material law). Enforcement is split honestly: the shim gates on reference.wav (the only engine-agnostic term); the pack builder is what will refuse to emit a pack without its artifact.
• Suite green: 384/384 (./tests/run.sh, wall 66 s). Landed as PR #4 (branch wt/20261017-181502 → main, merged as 49c4e19) — the first PR on this repo through the API, which cost three 422s to get right: the merge endpoint takes {"Do":"merge"}, Do being a string naming the strategy. DoMerge/merge_method do not exist on this Gitea; swagger JSON is disabled on the instance, so the 422 messages are the schema. Law saved to the gitea-api-pr-operations skill.
• Deliberately NOT in this phase: the :8880 engine decision and its speaker-artifact format, the pack builder, deployer/homepage/Prometheus registration, the models.yml row that lets omp call it, and open-notebook. Harness fact established while writing the PR body: the tts tool will dispatch an api: openai-speech provider row to POST <baseUrl>/audio/speech ({model, input, response_format, voice}), but the live speech.* vocalizer is Kokoro-enum-only — so "the agent talks in my voice" needs that provider row or a pi.registerProvider extension, decided in phase 2.

2026-10-18 — TTS engine picked: arm A qwentts.cpp, on the live-output criterion

• Frank listened to the four audition pairs (qwentts-*.mp3 against their armB-*.mp3 mainline partners) and passed every clone. That deleted the only axis the A/B could not measure, and the question was reframed: which engine is better for live output. On that criterion RTF only has to stay under 1.0 with margin — the discriminators are time-to-first-audio and run-to-run spread, because an arm whose RTF can cross 1.0 stalls a live stream mid-sentence.
• Chosen: arm A qwentts.cpp @ 5916bf7, the resident tts-server, 1.7 B talker Q4_K_M + Q8_0 tokenizer, in a frank-tts-engine container on 8880. The TTS_BACKEND=llama-tts-cli path in scripts/frank-tts.env stays in the repo as the rollback. The numbers that carried it: first audio byte 86–451 ms against a per-request model load, warm RTF 0.55–0.67 against 0.93–0.99, spread ±0.01 against ±0.25, and a native /v1/audio/speech against no HTTP path at all. Three consequences are configuration, not preference: boot-warm every voice, do not pin a cpuset (threads are hardware_concurrency()/2), register every pack with its WAV (a lone embedding is rejected).
• Two follow-ups deliberately not decided: the 0.6 B talker (0.44 RTF, 3.4 GiB, never auditioned — every clip is 1.7 B) and ICL vs embedding-only conditioning. Embedding-only is the shipped default because it is cheaper and needs no ref_text, which this fleet cannot transcribe honestly; .spk + .rvq still ride in every pack, so ICL remains a per-voice opt-in for batch work where total time beats first byte.
• Addressing law corrected in Local TTS and Voice Cloning: the pair may not meet at a host-loopback publish. netavark DNATs published ports only and a bridge container cannot reach a host 127.0.0.1 bind — the lesson that 502'd hermes — so host.containers.internal:8880, which scripts/frank-tts.env documents, is correct only for a routable bind. The deployed pair shares one quadlet network and the façade addresses the engine by container name.
• PR #12 (feat/open-notebook-podcast @ 114dd7a, 8 commits, 403 tests) is merged as 1eb8b21 (Frank's OK, given in session): open-notebook + SurrealDB quadlets on the private bridge 10.89.4.0/24, the API published on 5056 because seerr holds 127.0.0.1:5055, and ESPERANTO_TTS_TIMEOUT moved 600 → 660 to sit one round-trip outside frank-tts's own SHIM_TIMEOUT. Merged is not runnable: nothing listens on 7772 until the voice stack lands, and the episode destination (/home/dank/media/podcasts) stays interim pending Gitea #7.
• Still in flight, not in this branch: feat/frank-tts-engine — the arm A image build, the quadlet pair on one network, the boot-warm step, deployer / planner / test-changed.sh routing, homepage and docs. This branch is also where the two-branch overlap got resolved: merging main here conflicted on log.md alone (both branches appended an entry) and took index.md, deploy-remote.sh, deploy-plan.py and test-changed.sh clean.

2026-10-18 — open-notebook: the podcast generator's config lands (nothing deployed)

• Goal: open-notebook as the fleet's podcast generator on Dankclaw — notes and sources in, a multi-speaker episode out, script written by frank-zealot, voices rendered by frank-tts, no cloud provider anywhere in the path. Repo-side only: no host touched, no container pulled or started. Runbook: Open Notebook.
• Two containers, not one. lfnovo/open_notebook:v1-latest plus surrealdb/surrealdb:v2, both digest-pinned with Pull=never and a documented one-time podman pull — frank-dragoon's pinning style, because :latest moving underneath a rootless service is how you get an unbootable pod after an unrelated systemctl-daemon-reload. The registry round-trip is what proved v2 is SurrealDB 2.7.0 and that the app image is Debian trixie with curl and ffmpeg already inside — which is how both health probes can be argument-only: curl -fsS …/health (route is upstream's, api/main.py:239, and it sits on the auth exclude list) and /surreal is-ready, which I ran against the digest here: green in 2 s, and it refuses to pass without SURREAL_PASS in the environment — a credential that must never ride the command line, so Exec= carries flags only.
• Host 5055 was already spoken for. seerr has held 127.0.0.1:5055 since 2026-10-14 (live ss -ltn agrees today), so the API publishes 0.0.0.0:5056:5055. open-notebook's frontend derives its API URL from the window origin when API_URL is unset — the upstream env reference spells the reverse-proxy symptom — so a silent remap would hang the UI on a port nobody listens on, and API_URL=http://192.168.0.143:5056 is set accordingly. 8502 publishes on 0.0.0.0 because this service carries its own password (the episode-audio route is unauthenticated), unlike the loopback-and-proxied observability stack; 8000 is published nowhere, where upstream exposes it on loopback with root credentials, and the seam forbids it.
• The CPU guard, pre-armed. frank-tts spawns one ~8 GiB, 8-thread engine per request on a box that also serves two LLMs, while open-notebook defaults to TTS_BATCH_SIZE=5 × OPEN_NOTEBOOK_WORKER_MAX_TASKS=5 — a five-way self-DoS. Both are pinned to 1 in scripts/open-notebook.env, and the worker knob is only real because the image's own supervisord expands ${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5} (supervisord.conf:25). The TTS clock goes to 660 — one round-trip outside frank-tts's own SHIM_TIMEOUT=600, so a shim give-up reads as the shim's error rather than the worker's; the seam asserts the direction (>) against the frank-tts env instead of restating a number. ESPERANTO_LLM_TIMEOUT stays at the app's 180 s default: frank-zealot answers an outline or a transcript well inside it, and upstream says to keep it under 600 so a slow call fails with a name on it.
• What a credential needs in order to exist: OPEN_NOTEBOOK_ENCRYPTION_KEY must be set before the UI can save any provider credential, and OPEN_NOTEBOOK_PASSWORD must stay absent to remain auth-off — the app reads os.environ.get(…) is not None, so PASSWORD= boots a login form asking for the empty string. The repo env file carries both keys empty; the deployer seeds /home/dank/Containers/open-notebook/env (600) from the host vault and refuses to start the pair while either secret is blank, because a half-configured open-notebook stores no credentials and fails quietly.
• Episode destination is interim, by decision. The destination is Gitea issue #7, human-only — agents must not work it. Until Frank settles it, the app keeps its own named volume and /home/dank/media/podcasts is bind-mounted over /app/data/podcasts (DATA_FOLDER is hard-coded ./data), so finished MP3s are ordinary files in his home directory instead of volume contents needing podman unshare — the database volume demonstrably lands on a mapped subuid, measured with the pinned image. One mount line; one deletion undoes it.
• Two UI buttons will fail, and that is cosmetic. Test Connection does GET <base>/models and the TTS probe synthesises voice alloy; frank-tts serves /health and /v1/audio/speech only. Model rows are created by name and the voice field is free text, so narrator1/2/3 is typed in and never discovered. Documented, not worked around.
• Blocked on frank-tts, stated plainly: phase 1 landed without deployer or homepage registration and nothing listens on 7772 today, so a generation run writes its script and then fails on the first voice. That is upstream's creation-podcasts warning in fleet clothing.
• Suite green: 403/403 (./tests/run.sh, wall 67 s) — 19 tests in the new tests/open_notebook.bats, structure only: pins, boot persistence, the unpublishable database port, the 5055→5056 remap plus the API_URL that has to match it, the env-file read order, the guard knobs, the health gate naming /health, planner routing, the homepage card hidden while undeployed.
• Three deployer seams cost time and are now law-shaped: a service must be registered in five places — SERVICE_ORDER, DANKCLAW_SERVICES, the deploy-remote.sh dispatch case, its pre-deploy requirement, and deploy.sh's dispatch regex, where an omission silently plans an empty set for --only; a dankclaw/<svc>/* case must precede the dankclaw/*.container glob because that * crosses /; and the changed-path routing lives inside the per-path loop, after the scripts/*.env) case, not at the top of the file.
• Deliberately NOT done: the deploy (pulling 1.27 GiB to Dankclaw was not asked for), any Caddy vhost or public exposure, audiobook/EPUB (issue #5), any frank-tts code change, and the harbor templates on Dankserver — ~/.harbor/compose.opennotebook.yml pins the deprecated -single image, carries no encryption key and injects the whole harbor env — left untouched, read-only, as the record of what not to copy.

2026-10-18 — frank-tts-engine: the arm A engine becomes deployable (nothing deployed)

• Goal: turn the engine the day's A/B picked into a repo-owned, boot-persistent, self-healing container — frank-tts-engine, qwentts.cpp commit 5916bf7c79dbb4e66fc059cdc507a79a23a28bb4, resident tts-server, 1.7 B base talker Q4_K_M + Q8_0 tokenizer, container port 8880, and not a fleet endpoint: the endpoint stays frank-tts on 7772. Runbook: Frank TTS Engine.
• No PublishPort at all, which is the shape the merged A/B states as law. netavark DNATs published ports only, so a 127.0.0.1:8880 publish is not a path to a peer container — the façade could never have used it (the trap that 502'd hermes) — while being the one LAN-reachable surface of an unauthenticated synthesiser. Both voice containers therefore sit on one committed quadlet network, frank-tts-engine.network at 10.89.5.0/24 (the next free /24 after open-notebook's 10.89.4.0/24), and the façade addresses the engine by container name. Diagnostics moved to podman exec, including the deployer's health gate (wait_container_healthy, not wait_for_health: there is no URL to dial).
• The boot warmer is a unit hook, not an operator habit. Cold-voice law: the first request for a registered voice decodes the whole reference to snapshot codec state (6.2–8.0 s, first byte 5 530 ms), later ones restore in 0.5 ms; the registry is memory-only and re-registering does not warm. So ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh with TimeoutStartSec=900, and the - prefix is what stops a failed warm from crash-looping the endpoint under Restart=always. It logs cold= and warm= per voice, so the under-a-second second request is proven on every boot.
• Thread law recorded before anyone tunes it away: no --threads exists; src/backend.h uses hardware_concurrency()/2, so a cpuset pin silently changes the thread count. Nothing pins this container.
• Image builds from the pinned commit with the proven line, verbatim — GGML_BLAS=OFF is forced (this host has libopenblas.so.0, no dev headers), every GPU backend off, -j 8, and no -DGGML_NATIVE because ggml's default is -march=native for the building host: the image is built where it runs. Base ubuntu:24.04 pinned by index digest; two stages, because ggml builds shared and the RPATH names the build tree.
• Weights fetched by scripts/download-frank-tts-engine-qwentts.sh with byte and LFS-oid checks (talker 1 219 245 248 B, tokenizer 291 150 624 B); the deployer gates on --verify-only rather than pulling 1.4 GiB inside a deploy.
• Registered in all five places the open-notebook entry just named as law, plus the routing case above the dankclaw/*.container glob. New seam tests/frank_tts_engine.bats (18 tests): the Quadlet generator's own output for the network/publish/boot/warm shape, the entrypoint's argv driven from the tracked env file, the warmer against a stub tts-server over a real socket (registration keys, ref_text absent, late engine, hung synthesis, name refusal), the fetcher's verifier, and both routing tables.
• Deliberately NOT done: the deploy (no host touched, no container started, no image built), any homepage card or Prometheus job for the engine — it publishes nothing and exposes no metrics, so there is nothing to probe, which is said out loud in the page rather than papered over. No ICL/ref_text work: the packs keep reference.spk/reference.rvq so it stays a per-voice opt-in.

2026-10-18 — frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down

• Stage 2 of the voice stack: the façade in front of the engine is now a deployable service too. Nothing deployed, no host touched. Runbook: Frank TTS.
• Two attached networks, in a meaningful order. dankclaw_default first — it owns the default route and is where consumers live (open-notebook dials http://frank-tts:7772/v1 by container name) — then the voice pair's private bridge. scripts/frank-tts.env's TTS_UPSTREAM moves from host.containers.internal:8880 to http://frank-tts-engine:8880, which is the actual fix for a shape that could never have worked: a bridge container cannot reach a host-loopback bind, and the engine publishes nothing.
• Wants= + After= on the engine, never Requires=: the façade still carries the llama-tts-cli rollback backend, which needs no upstream, so an engine that will not boot must delay this endpoint, not take it down.
• The façade handler installs the engine's quadlet pair as well. Not tidiness: with a referenced Network= file absent, the v1 Quadlet generator produces no container unit at all, and --only frank-tts has no other way to obtain it. It also mkdir -p /home/dank/voices as the deploy user, because rootless podman auto-creates a missing bind source root-owned and the endpoint then reports "no voices" instead of an error.
• A voice is a lowercase directory name. ^[a-z0-9][a-z0-9._-]*$, ≤ 64 chars, no slash/../space. The shape is the intersection of two rules that do not agree: app.py would accept mixed case, tts-server lowercases names — so a pack named Narrator1 resolves through the façade and not through the engine, and the boot warmer refuses such a directory loudly. narrator1/2/3 are load-bearing for open-notebook, and frank-tts-add-cloned-voice is the skill that walks the pack → register → warm → prove-a-second-request flow.
• Registered everywhere the open-notebook entry named as law (planner order puts the engine before the façade, so a warm registry precedes the first synthesis), plus the homepage card — wip: down and hidden, because nothing listens on 7772 yet.
• No Prometheus job, on purpose. The façade exports no /metrics, Prometheus 3 refuses HTML/JSON content types (the comfyui and services probes are documented down for exactly that), and LlmEndpointDown is scoped up{job=~"frank-.*"} — so a /health scrape would page on Signal every few minutes for a healthy endpoint. That is the frank-archon never-green lesson pointed the other way, and it is written in the page instead of being quietly ignored.
• Tests added to the voice seam: the generator's own rendering of the pair (both attachments, first-network order, one publish, Wants-not-Requires), the env-file upstream checked as a network join rather than a string comparison, planner separation and order, the façade handler's steps, the homepage card's gating, and routing that selects both seams for either deployable's paths.

2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code

• Third slice of the voice stack (after the façade and the engine decision): scripts/build-voice-pack.sh turns a source recording plus a crop offset into a deployable pack — reference.wav (24 kHz mono pcm_s16le, 18–25 s), the engine's own reference.spk (8,192 B) + reference.rvq (5,456–6,864 B) made from it, and manifest.json provenance. Tracked tunables in scripts/voice-packs.env: pack list, per-voice source/offset/duration/pinned source sha256, extractor path and engine commit, artifact arithmetic. New runbook Voice Packs.
• The gate is the product. The project-wide invariant — no voice material from the first 10 s of a sample file — had only ever been a sentence in scripts/frank-tts.env. An offset inside it now exits 1 before a directory is created, and the same assertion runs again against the manifest that was just written, next to the recorded sha256, so a pack cannot carry a forbidden crop even if the env file is edited into nonsense. 10 stays a constant in the builder; the env file carries offsets, never the law.
• The extractor's contract was read, not guessed: qwen-codec in qwentts.cpp 5916bf7c takes --model <codec gguf> --talker <base talker gguf> -i <wav> with an optional --device, has no -o and names both artifacts next to the input with the extension swapped. So the WAV is staged under its final stem and reference.wav — the file the façade gates on — is moved into place last, after the artifacts and the manifest: a failed extractor leaves a directory that is not a voice. --force retires an existing pack whole to .AGENTTRASH/ instead of touching bytes; nothing in this script deletes.
• Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B per 12.5 Hz frame (22 × floor(samples/1920)) and enc_dim 2048 × f32 = 8,192 B. That makes the effective crop window 19.84–25 s, narrower than the WAV law's 18–25 s, and a too-short crop is refused before the extractor runs. An engine bump that changes either number fails the build by design; the bump procedure is in Voice Packs.
• Verified against reality, not against the model: the pinned narrator1 crop (98.739–118.636 s of audiosample.mp3, sha256 2299510…) comes out of a --dry-run at 955,378 B / 19.897 s with 5,456 B of .rvq planned — byte-for-byte the trial's hand-cut numbers — and all three pinned source digests match the files on disk.
• Seam: tests/voice_pack_unit.bats (13 tests) — refusal inside the forbidden region creates nothing, a legal crop builds the whole pack, ffprobe on the generated reference, manifest round-trip through jq with a sha that matches the source, re-run-without---force leaves every byte alone, --force retires rather than deletes, --dry-run needs no extractor, a wrong-sized artifact from a bumped engine cannot publish a voice, and every crop in the tracked pack list clears the invariant. qwen-codec is stood in by a stub with the real tool's I/O shape, so no engine, no GPU, no network, no fleet host. Suite green: 416/416 (./tests/run.sh, wall 77 s) — 403 before this branch.
• Deliberately NOT done: no transcript in the manifest (ICL wants a curated ref_text; the trial's machine transcripts were wrong, and the engine registers WAV-only voices anyway), no 16 kHz variant (arm A is native 24 kHz), no loudness pass (not in the pack law), and no pack built on the host — the three sources still live on the workstation and the engine GGUFs have not been fetched to /home/dank/models/Qwen3-TTS-GGUF yet, so the first real run is a host step.

2026-10-18 — voice stack audit: three faults that no gate could see

• The voice stack had never been run end to end — no image built, nothing on 7772 — so the whole path was audited as static code against the tools it calls, and three faults came out of it. All three are the same species: a step that reports success while doing nothing.
• The build would not have happened. Containerfile.engine created its unprivileged user with adduser; the pinned ubuntu:24.04 base has no adduser at all. Proven in the base on Dankclaw rather than argued from memory: command -v adduser prints nothing, and the replacement line useradd --system --uid 10000 --home-dir /app --shell /usr/sbin/nologin tts answers uid=10000(tts) gid=999(tts). Its SYS_UID_MAX warning is cosmetic and is the price of matching the façade's uid, so the comment says that before someone "fixes" it by dropping --system.
• The warm pass would not have warmed. The unit hands its env file to podman via --env-file, which performs no shell parsing and takes the literal remainder of the line, while source strips one quoting layer. WARM_TEXT therefore reached the container with its quotes, the hook interpolated them into the synthesis body, the JSON was invalid, and every voice would log "registered, but the warm synthesis failed" beside an active unit. Fixed in the hook — strip exactly one matched pair, only when the inner content also carries a quote or backslash, then JSON-escape — and the seam now reads scripts/frank-tts-engine.env both ways and asserts the engine gets the same sentence in both shapes. The bug survived the first deploy precisely because every existing test used source.
• A rebuilt image would not have been rebuilt. build_compose_image skips podman build on a matching cfhash label, and that hash covered the Containerfile alone — so editing app.py or tts-engine-warm.sh (both COPY'd) skipped the build, the deploy reported success, /health answered, and the container served old code. Each case arm now names the files its Containerfile copies and hashes them with it; comfyui/loki/alloy have no extras, so their labels are byte-unchanged and they do not rebuild here.
• Tests as behaviour, not as greps: the new seam extracts the real build_compose_image from the deployer and drives it against a stub podman that replays per-image label state, asserting the observable question — did the deploy ask podman to build, or did it skip? — through build, skip, engine-hook-edit-rebuilds-engine-only, and app.py-rebuilds-the-façade. Reverting scripts/deploy-remote.sh turns it red at the warm-hook step, which is the fault being prevented. Suite green: 443/443 (./tests/run.sh, wall 110 s); each fix commit was left green on its own so the branch bisects.
• Docs carry the lessons where an operator will hit them: a Locally built images section in deployment, two new troubleshooting lines in Frank TTS Engine (all-voices-cold = quoted WARM_TEXT; façade 502 beside 200 = model loaded ≠ voices registered), and Voice Packs.md step 4 corrected — the warm hook re-registers every pack on every run, because the engine replaces a name it already holds (tools/tts-server.cpp:148 at the pinned commit: "Re-registering a name replaces the previous entry."), so a restart is what an engine bump needs and not what a rebuilt pack needs.
• Opened as PR #14 against main. Deliberately NOT done: no deploy (the first --service frank-tts-engine run is a ~10-minute recursive source clone and a new unit on a host that serves people — Frank's call), no voice pack built on the host, no Playwright run (this branch publishes no host port and touches no vhost), and Gitea #2 phase 1 stays open until 7772 answers.

2026-10-18 — frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap

• Merged as #14 (747c665) and deployed --service frank-tts-engine. The payload took 18 s to fetch for the 1.4 GiB pair (byte count + Hub LFS sha256, --verify-only OK), and the image built clean — the first real proof of the useradd fix: STEP 3/13: RUN useradd --system --uid 10000 … passed with only the documented cosmetic SYS_UID_MAX warning, and the unit was healthy after 31 s.

• Then the deploy reported 0 voice(s) warm: /app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long. The hook handed the registration body to curl as ONE argument; the kernel caps a single argument at 131,072 bytes (MAX_ARG_STRLEN) and a shipped 955 KB reference.wav base64's to about 1.27 MB, so curl never executed and the hook logged that as "HTTP no response". Fourth silent fault of the voice stack, same species as the other three: /health stayed green throughout.

• Fix: the body goes to a mktemp file and curl gets @file. The regression test is the honest shape — the seam's stub engine is a real listener, curl is the real binary, the fixture pack is 300 KB — so it fails with the same kernel error the host produced (proved red first, green after). Every shipped pack sits well over the ~96 KB cliff, so no small fixture could ever have seen this.

• Live proof on the deployed container with the three real packs: narrator1 cold=1175ms warm=1874ms, narrator2 1373/1892, narrator3 1816/1487, exit 0, 15.2 s for the whole pass.

• A trial law did not survive deployment. Against an empty registry — register only, then time — the FIRST synthesis of a freshly registered voice cost 1 883 ms, the next two 1 283 and 1 226 ms; a ~110-word request cost 9.4 s then 9.8 s. The trial's 6.2–8.0 s cold / 0.5 ms restore ladder, which the warmer's own header repeated, is retired: warming buys registration — mandatory, because the registry is memory-only — plus about 0.6 s. Corrected in Frank TTS Engine law 1 and its troubleshooting table, in Local TTS and Voice Cloning, and in Voice Packs.

• Host state, honestly: the engine is active with three voices warm, but the hook fix is not in the image yet, so the next restart re-breaks it until this branch is merged and re-deployed. frank-tts (the 7772 façade) is still undeployed.

• Deliberately NOT done in this window: the frank-tts deploy, any Prometheus probe for the voice pair (neither container exposes /metrics; the blackbox question is still open), and the podcast render — open-notebook is on main but not installed.

• Sweep afterwards: the retired ladder still lived in six places the first pass missed — both quadlet files, scripts/frank-tts-engine.env, scripts/frank-tts.env, the Frank TTS troubleshooting row, the research brief's consequences paragraph, and one test comment. All re-keyed to the measured claim: about 0.6 s, and the 400-for-every-name reason the pages should have led with. The trial entry above is left as written — a dated worklog entry is a record — and this line is the correction that points at it.

2026-10-18 — 7772 is live and the harness speaks through it

• Merged #15 (0315a62, the warmer's argument-cap fix) and #16 (d2ce669, the comment sweep), then deployed both voice services in order: --service frank-tts-engine (image rebuilt from the fixed warmer, unit active, zero drop-ins, warm pass in the unit's own start — narrator1 cold=1542ms warm=1709ms, narrator2 1329/1405, narrator3 1451/1394, 3 voice(s) registered and warm) and --service frank-tts. The Playwright gate ran at the end of each deploy: 33/33 twice.
• Deployed-state proof from a LAN client: /health → omni-http with all three voices; the same sentence in narrator1 and narrator2 came back with different sha256 and different RMS (4.24 s, 24 kHz mono PCM16), so the name actually selects a voice. Engine-side ledger for the harness request: TTFA 67.5 ms, Total 2158.8 ms, audio 3.76 s, RTF 0.574.
• The harness speech role moved off Kokoro: config.yml modelRoles.speech: frank-tts (dankclaw)/frank-tts. No voice key moves with it. The bundle is unambiguous: the speech device forwards the call's voice_id and nothing else, tts.localVoice belongs to the local backend (it is what answered in the control test), and speech.voice is the TUI's speak-aloud enum.
• Two harness laws learned by breaking it first (both now on Frank TTS): an explicit speech role has no fallback candidate, so a dead endpoint errors instead of degrading; and models.yml loads at process start while config.yml is read live, so a session older than the frank-tts row fails exactly like a dead endpoint. Fresh-process proof in the same minute: Saved 180524 bytes (model=frank-tts (dankclaw)/frank-tts, voice=narrator1) with the matching façade log line speech voice=narrator1 format=wav chars=51 -> 180524 bytes. This window's own session is that stale case and knows it.
• Status sweep, because six pages still said the endpoint did not exist: both vault descriptions, the two index.md bullets, the Open Notebook blocker (the voice leg works; open-notebook itself is what is missing), the engine page's "Deploy order — nothing below has been run" is now a Deploy record, and the homepage card lost wip: down so the admin panel probes 7772 for real.
• Corrected the same evening, by breaking it in a restarted session: a harness call that names no voice reaches 7772 with no voice key at all — 400 unknown voice None; available voices: narrator1, narrator2, narrator3 — because no setting feeds one. The earlier Saved 180524 bytes (voice=narrator1) succeeded only because that run was told to pass voice_id. The endpoint therefore grew DEFAULT_VOICE (its own PR): a voice-less request speaks the declared pack and the journal line carries (default), while an unknown name still 400s with the list. Reverting the role to Kokoro is one key, not two, and speech.voice: narrator1 — briefly recorded above as wiring — went back to am_fenrir: that key is the vocalizer's Kokoro enum, so a pack name there breaks speak-aloud and does nothing for synthesis.

2026-10-18 — the default voice shipped: a harness voiceover needs no voice

• Merged #17 (b590695) and #18 (7889a68), then deployed both affected services in one dispatch — --service frank-tts --service homepage, 159 s, Playwright gate 33/33. The shim image rebuilt with the new handler; the homepage checker took the card on its next timer tick: frank-tts → up, probed for real rather than gated.
• The call that had failed is the one that proves the fix. Same device, same restarted session, no voice_id: Saved 430124 bytes (model=frank-tts (dankclaw)/frank-tts, voice=default, codec=wav). The façade journal says what actually spoke — speech voice=narrator1 (default) format=wav chars=148 -> 430124 bytes — with the engine's ledger behind it (audio 8.96 s, RTF 0.551). The harness prints voice=default because it never had a name to print; the voice that answered is the operator's line in scripts/frank-tts.env, and the journal keeps them distinguishable forever after.
• The page gained a Default voice row in its contract table and #Naming a voice now states that a call naming nothing is the harness's normal shape, not a client bug — the misconception that produced the 400 in the first place.

2026-10-18 — OpenWebUI "voice narrator1 not found": the rejector is in the browser

• Frank's toast — Error: Voice "narrator1" not found. Should be one of: af_heart, …, bm_fable — looked like frank-tts refusing the name. It is not: that 29-name enum is the voice pack of Kokoro.js running in the page, and OpenWebUI 0.11.4 ships that implementation in its own front end (frozen table {af_heart:{name:"Heart",…}}, _validate_voice, model onnx-community/Kokoro-82M-v1.0-ONNX, a personal Kokoro.js dtype setting, engine values ""/browser-kokoro).
• Instruments that separated the two paths, in order: the admin config DB is correct (audio.tts.engine=openai, audio.tts.openai.api_base_url=http://host.containers.internal:7772/v1, model=frank-tts, voice=narrator1, split_on=punctuation); from inside the container the exact OpenWebUI body ({"model":"frank-tts","input":…,"voice":"narrator1","response_format":"mp3","speed":1.0}) returns 200 audio/mpeg 7,629 B; and podman logs open-webui holds zero audio/speech requests for the failing click. A server-path failure leaves a log line and a frank-tts error — a browser-kokoro failure leaves neither. No container env carries an AUDIO_*/TTS variable and no user.settings row (all four users) carries an audio override, so the engine choice is per-browser storage, not fleet state.
• Fix is a UI click, not a deploy: Settings → Audio → TTS Engine → Default hands synthesis back to frank-tts and the cloned voices; choosing af_heart keeps in-browser Kokoro, which cannot clone (fixed pack). Recorded on OpenWebUI under Text-to-speech (frank-tts); suite green 384/384 (wall 65.9 s).
• Also in the window: PR #22 merged (0c7fe8b, dank/command-center) so agent/extensions/dankvoice.ts landed on the auto-load path — proven in a plain omp session with no flags: /say merged build check → 1.20 s WAV to the player, temp dir cleaned; the audible pass through the default pw-play detection confirmed by Frank. Sessions open before the merge need a restart to pick the command up.

2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV

• Frank's instruction: "shift frank-zealot to 2 parallels of unquantized kv". The env delta is four lines: CTX_SIZE=524288, PARALLEL=2, CACHE_TYPE_K=f16, CACHE_TYPE_V=f16. Everything else holds — batch 4096 / ubatch 2048, --flash-attn on, --jinja, --metrics, --presence-penalty 1.0, SPEC_TYPE=none, the Q8_0 mmproj, port 7771, and the per-slot window 262,144 — models.yml contextWindow untouched.
• Why the pool had to halve. Exact KV bills 32,768 B/token (8 full-attention layers × 4 KV heads × 256 dim × 2 B × 2 caches), so the old 786,432 pool asks 24,576 MiB against the 24,123 MiB the driver leaves addressable — impossible, not tight. 524,288 positions ⇒ 16,384 MiB and exactly two windows of 262,144.
• Arithmetic correction to the repo's long-standing figure: "the f16 pair would need 26,112 MiB" doubled the q8_0 number including its 1/32 scale overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is 1.882×, not 2×, and exact f16 over 786,432 is 24,576 MiB. The conclusion stands (three f16 windows cannot boot this card); the number was wrong.
• The cost: concurrency 3 → 2 conversations, and MTP is now doubly out. The 13:16 measurement still says off, and exact KV adds a second reason — the draft context bills its own f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) on a trunk projecting ~21,200 used.
• Projection, explicitly not a measurement: ~21,200 MiB used / ~2,920 free, derived from the 13:19 ledger's 17,922 used because only two terms move — the KV pool (+3,328) and one recurrent-state cell (−50.25, 3 cells → 2). The measured boot ledger and the decode/prefill re-measure land in the follow-up commit, after the first f16 boot.
• New gate in tests/frank_zealot_unit.bats: dtype × pool plus the measured non-KV terms must fit 24,123 MiB with ≥ 1 GiB of slack — and the same gate proves it refuses the impossible arm (f16 at 786,432), not merely passes the shipped one. Runbook re-keyed to the new shape: LLM/frank-zealot-llama-prism-bonsai9-mtp.md.