# Bundle Change Log

## 2026-10-18
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
  `--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so
  both remaining conversations keep the native 262,144 window and the compaction
  hub reads exact K and V. **Halving the pool is what funds exact KV** — the law
  that came out of it is that a KV byte rate belongs to the payload it was
  measured on: the doc's old "f16 needs 26,112 MiB and cannot boot" figure was
  the 27B's 34,816 B/token applied to the 9B. The measured boot ledger is on
  [[frank-zealot]] (Current shape), which is also where the 32,768 B/token
  derivation and the seam's fit gate live.
  - **Rates are floors, and that is recorded as such.** The measurement window
    had another consumer on the other slot (56 prompts in three minutes, 96 %
    GPU-util) — the exact condition the bench window exists to exclude — so every
    rate taken that evening is recorded on [[frank-zealot]] as a **FLOOR**, not as
    the shape's rate, with the instruction to re-measure with the box quiet before
    a figure is carried anywhere. That is the honest shape of a number measured
    beside live traffic.
- **Instrument correction (fleet law).** On the PrismML fork build 10777 the raw
  `/completion` path answers **500 "The model produced output that does not
  match the expected Content-only format"** for a greedy `ignore_eos` run, so
  the honest per-request `timings` for this endpoint must be read off
  `/v1/chat/completions`. `AGENTS.md` said `/completion`; it now says the chat
  path.

## 2026-10-13
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
  power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
  from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
  below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
  preflight named the refusal and exited 1, and `Restart=always` re-ran it
  **6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
  container, no engine log. Law adopted: **a sysfs raise is a countdown to the
  next reboot**, so the gate raises the knob itself. New
  `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
  re-verify, then refuse by name) and is also the unit's
  `ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
  and the knob is root-owned 0644, which is what the first attempt (a plain
  write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
  records the host ceiling.
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
  `--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
  measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
  at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
  to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
  fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
  `models.yml` needed no change: the per-request window is unchanged, only the
  concurrency.
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
  backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
  sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
  of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
  (now reading the shape from the deployed env): four concurrent 196,960-token
  chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
  aggregate prefill, no preemption; prefix cache verified (repeat answered in
  1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
  homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
  4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
  was funding 858,969 tokens — 43 % over the contract. Then slab
  24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
  OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
  96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
  real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
  invented on 2026-10-11. The pair now spends 30,720 and settles
  `unclaimed 477.30 MiB` (cards ~336 MiB free).
- **The prediction shipped with that arm was falsified by its own measurement,
  which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
  below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
  other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
  cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
  193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
  pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
  points inside noise of each other are not a plateau. The cliff is above
  25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
  slots (26,112 + 5,120 > the ceiling) and free at two.
- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
  accepted and completed (599,952 tokens resident), ledger
  `backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
  law's 608,027. A fourth full-height request is refused as too large to ever
  fit — radiance never truncates. Probe law: the oversized-prompt counter is
  FREE (the refusal fires when PROMPT tokens alone exceed the window —
  `prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
  window is ~514,286 characters at the measured 2.5714 chars/token, so a
  calibration probe below that pays a real prefill instead of reporting.
- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
  exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
  conversation. Untested arm; capacity decision, not tuning.
- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
  owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
  measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
  own test. 371/371 tests and e2e 33/33 on both deploys.
- **Power-failure casualties, NOT fixed (outside the archon window):**
  `mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
  raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
  media libraries on it read empty. `frank-job-digest.service` (frankbook
  job-hunt digest) is in `list-units --state=failed`.
- **The slab is a threshold, not a slope** — measured, same method (probe A
  leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
  22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
  bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
  (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
  would clear it (pool 3,072 + slab 27,897) remains untested — and the
  paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
  not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
  cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
  is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
  give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
  The slot count is paid in decode concurrency — a fused-step lottery, not a
  fair share — which is exactly the axis the shape hunt was blind to (probe
  leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
  is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
  table, so these rows are the baseline for this corpus, not a replacement.
- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
  tokens, per-stream + aggregate) so no future shape change can be priced on
  prefill alone.


## 2026-10-10
- **Fleet service rename: the StarCraft names became the SERVICE names.**
  The 2026-10-09 wire alias moved onto every repo artifact: units, serve
  scripts, env files and their `FRANK_*` variables, deployer service keys,
  container names, Prometheus/Grafana job names, bats seams, runbook
  filenames and all historical prose. `frank-archon` (dankserver :7770),
  `frank-dragoon` (dankclaw :7770), `frank-zealot` (dankclaw :7771). The
  three former service names are fully retired: `git grep` over the tree
  returns zero, rollback tags were renamed in lockstep (e.g. tag
  `frank-archon-freetoken-pr587-20261005`), and the harness `models.yml`
  lore matches. Host cutover = stop old unit + remove old container,
  deploy the renamed service (the deployer relinks the unit under the new
  name; `add_pending_services` auto-trashes the retired markers). The
  retired `frank-medium`/`frank-small`/`frank-npu` names are historical
  and stay as they are.
- **Known casualty:** the two vendored vLLM-fork patches under
  `scripts/patches/` carry the old service names in context lines; the
  sweep rewrote them, so those patches (retired eras, rebuildable from
  the renamed tags) no longer apply byte-exact.
- **Executed live at `55f87fc`, all three gates green.** Per host: stop
  old unit → trash its deploy-state marker + remove the unit symlink and
  old container → `deploy.sh <host> --service <new-name>`. Result:
  `frank-archon` active/enabled (container `frank-archon`, :7770),
  `frank-dragoon` active/enabled (container `frank-dragoon-halogen`,
  :7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving
  compaction on :7771). `/health` 200 ×3; `/v1/models` reports the three
  ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky,
  33; Prometheus `frank-archon`/`frank-dragoon` up.
- **Pre-existing gap found by the verification, NOT a rename regression:**
  `frank-zealot` `/metrics` answers 501 — llama.cpp gates the endpoint
  behind `--metrics` (default disabled) and the serve script never passed
  it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h
  (the zealot metrics + the queue-depth alerts on it have been dead since
  the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min
  zealot restart; not applied in the rename window.

## 2026-10-09
- **Live — five authentik human accounts provisioned (Dankclaw, `ak shell`
  ORM, no repo objects):** kirby (pk 17), ry (18), nic (19), debo (20),
  freya (21), each `<user>@franksdank.cloud` in the human-user template shape
  (`is_active=True`, password via `set_password` on stdin only, groups
  `Non-Admin Users` + `jellyfin_users` + `openwebui_users`). Verified through
  the exact PasswordStage code path (`authenticate()` accepts each password and
  rejects a wrong one) and `app_entitlements(open-webui)` → `Open WebUI Users`
  for all five (identical to the ridley precedent). No `proxysession` rows
  existed for them — groups were set before any login, so no claim purge was
  due. Downstream: Jellyfin visibility rides the LDAP outpost cache (expect
  lag); OpenWebUI mints their accounts as role `pending` on first login.
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
  census extended with pk 17–21.
- `/serverAppConfigs-docs/Dankclaw/OpenWebUI.md` — new SSO bullet: first login
  lands as role `pending` (`ui.default_user_role=pending` in the config DB),
  the role-flip SQL, and the global `access_grant` (`principal_id='*'`) fact
  that makes the picker model visible to every `role='user'` account.

- **Live — the Protoss rename finished on the wire (all three endpoints).**
  Served model ids moved `frank-archon` → **frank-archon**
  (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` → **frank-dragoon**
  (`HALOGEN_MODEL_ID`), `frank-zealot` → **frank-zealot** (`ALIAS`). The
  services, units, scripts, env files and runbook pages keep the old names —
  container/unit identity stays stable, only the alias moved. Deploys in order:
  `dankclaw --service frank-zealot` (e2e gate 33/33 green), `dankclaw
  --service frank-dragoon` (healthy 30 s), `dankserver --service frank-archon`
  (healthy 102 s), `dankclaw --service homepage`. Verified on `/v1/models`,
  not on chat echoes: `frank-zealot` (`n_ctx 200704`, Q6_K), `frank-dragoon`
  (`context_length 262144`), `frank-archon`.
- **Live — OpenWebUI re-keyed in the same window** (snapshot
  `webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`,
  params + `meta.description` preserved), the `access_grant` row
  (`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs`
  `model_ids` on all three backends; `systemctl --user restart
  open-webui.service`. Family-account proof, in a real browser as `qa-e2e`:
  the welcome screen selects `frank-zealot` and
  `POST /api/v1/chat/completions {"model":"frank-zealot"}` answers
  `zealous` with `"model":"frank-zealot"`. A `GET /api/v1/models/list`
  probe returns `{"items":[],"total":0}` for that account while the picker
  is plainly populated — the API census is not the acceptance test, the UI
  snapshot is.
- **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon`
  in the provider block, `model.default` and the three `dankclaw-large`
  auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` +
  `hermes-webui.service` restarted, gateway log clean, `/health` ok) and the
  repo baseline `dankclaw/hermes/config.yaml.example` synced to the live
  shape (context 262144). **Flagged, not fixed:** Hermes still routes its
  `dankserver-small` auxiliary tasks at the id `frank-medium` on
  `192.168.0.235:7770`, an endpoint that has served `frank-archon` since
  2026-09-20 — those routes were dead before this rename and remain dead;
  they need a routing decision (archon capacity), not a rename.
- **Deployer blind spot found by accident:** every
  `deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09
  restarted the **staging worktree**, not the deploy checkout — a campaign
  drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and
  `ExecStart`. The health gate went green each time. Verified parity before
  removing it (serve script + unit byte-identical, env identical except the
  alias, staging HEAD an ancestor of `main`), moved the drop-in to
  `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and
  restarted: `/v1/models` then answered `frank-archon`. Post-deploy
  behavioural proof is now part of the runbook.
- Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon`
  unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`,
  `calib-push`, `fleet-probes-swift15`, `autoresearch*`,
  `bench-threads-batch-http`, `bench-decode`), `shared/homepage/docs.html`
  now teaches the ids it ships (curl/opencode/Pi send `frank-zealot`),
  `AGENTS.md`, [[OMP Harness]], [[Homepage]], [[OpenWebUI]] and the three
  endpoint runbooks updated. Suites: `./tests/run.sh` 366 green.
- **Transcript secret incident:** dumping `podman exec hermes-webui env`
  printed the Porkbun API key + secret key into this session's transcript —
  same class as the 2026-09-16 HF-token-in-`ps` leak. Not committed
  anywhere; rotation is Frank's call.

## 2026-10-08
- **frank-archon (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 —
  the radiance quantization campaign.** ukisai's BF16 checkpoint (335.3
  GiB, 397/397 sha256-verified) → `qwen4exp-swift15-w4dn8-i8-hc8m.rad`
  (~114 GiB): all 49,152 experts GPTQ w4 (act-order, damp 0.05) calibrated
  on 10,000,000 live-model tokens, 24 whole bf16-protected experts per
  layer derived from the calibration grams' energy share, int8 KV, fp8 HC
  mixtures. Engine pins: **1.0.13** for serve/convert/KLD, **1.2.0** only
  for calibration (the sole pin exposing `RADIANCE_CALIB_TOKENS`; the
  RADGRAM2 gram format is byte-identical across pins, so grams cross the
  boundary by design). Chain scripts in `scripts/`:
  `make-calib-corpus.py`, `run-swift15-bf16.sh` (bf16 reference: KLD record
  + calib tap), `calib-push.py`, `derive-protected-experts.py`,
  `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`,
  `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`.
- **Engine laws learned (measured, not inferred).** (1) The qwen4exp_fp8
  plugin declares **one encoding per expert table**: a recipe may NOT split
  `ffn_gate_up_exps` (w4) from `ffn_down_exps` (block-fp8) — the serve
  kernel is a single w4nl64a8h table for the whole expert set; the only
  legal beside-table content is whole bf16 experts (the PROTECTED section).
  (2) w4 expert tables stream from the file tier **in place only at
  MNBT ≤ 2048**; at MNBT 32768 the engine demands a widened form and the
  disk tier refuses (`DID NOT FIT … stored form is not the container's`).
  (3) The calibration tap lives in the architecture plugin
  (`/opt/radiance/share/radiance/architectures/qwen4exp_fp8.so`), not the
  binary — `grep -ac` the `.so` before declaring a flag missing.
  (4) Disk-tier boots need ~9.6 GiB of weight buffers + arena charged to
  the VRAM weights slab; explicit `--vram-weights-mib`/`--vram-kv-mib`
  pairs replaced the fp8-era EVC ratio.
- **Throughput gate PASSED with 4.5× margin** (campaign boot, TP=2, int8
  KV, MTP 3): decode **136.0 t/s median** (122–173 across probes) at
  44.5k-token depth vs the 30.8 t/s bar; MTP acceptance 0.725; warm TTFT
  9.45 s; prefill 6.4k t/s. The fp8-native control on 1.0.13 measured
  30.08 t/s decode, and the 1.2.0 engine re-baseline was REJECTED
  (29.21 t/s, acceptance 0.7123 < 0.7549) — serve stays on 1.0.13.
- **Deep-output audit:** a 44.5k-token verbatim-recall probe enumerated all
  48 planted documents in order — the DDR5-corruption signature is absent.
- Fleet probes (`scripts/fleet-probes-swift15.sh`): tools, reasoning
  levels x4, reasoning_content, response_format, vision — all PASS.
  KLD gate vs the BF16 reference: recorded on 1.0.13 (50,235 positions,
  197 prompts), scored on the same pin at the w4-legal shape.
  Verdict: see runbook [[frank-archon]].

## 2026-10-06
- **frank-zealot (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
  RTX 3090).** "set ubatch to 2k" → commit d8da7a9: `UBATCH_SIZE`
  1024 → 2048 at the 2×114,688 shape. **Cost 746 MiB**, not the 616 MiB
  the same rung measured on the MiMo shape on 2026-10-05 (a rung's price
  is shape-dependent; the 616 figure was not transferable). Boot:
  **23,120 used / 1,005 MiB free**; a 512×512 vision probe then took
  ~240 MiB **and kept it** — settled **763 MiB free**, the encoder's
  buffers staying resident across requests on this build, unlike the
  evening boot where the 34 MiB transient returned. Decode 52.0 t/s on
  the shared prompt (58.2 at the 1024 rung) with acceptance 34.2 % vs
  42.8 %: ubatch does not move decode, so the gap is temp-1.0
  acceptance noise, not a regression. **Prefill 1,209 t/s @ 3,760
  tokens** — the first prefill number this endpoint has recorded, and
  now the baseline future rungs compare against. `BATCH_SIZE` stays
  4096, so the seam's batch ≥ ubatch guard holds; **2048 is the ceiling
  for this shape** (4096 = no micro-split doubles the buffer growth
  again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
- **frank-zealot (Dankclaw): RENAMED from frank-small, then reshaped
  to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit
  87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers
  `FRANK_ZEALOT_*`/`llm_zealot`/`deploy_llm_zealot`; historical
  refs to the OLD FreeToken/CUDA frank-small and to the Dankserver
  service that became frank-medium stay frank-small, as does
  `freetoken-frank-small.md`). Live unit transition by hand before the
  deploy: `systemctl stop frank-small` + `systemctl disable frank-small`
  (a relinked unit — disable removes only its stale symlink) →
  `deploy.sh dankclaw --service frank-zealot`. The reshape (commit
  f2575a1, Frank's order "lower max context to 112k so we can increase
  ubatch to at least 1k"): `CTX_SIZE 262144→229376` (per-slot
  131,072→114,688), `UBATCH_SIZE 512→1024` → **22,374 used / 1,751 MiB
  free** (the per-slot cut frees ~1.05 GiB of KV; ubatch 1024 spends
  part, netting ~1.25 GiB back over the 489 MiB fit), decode 58.2 t/s
  recipe / 66.9 temp 0 (acceptance 42.8–54.4 % — the drop vs the prior
  boot's 63.3/72.9 is confounded by prompt text, not a shape verdict).
  Harness retargeted: models.yml (id/name/compactionModels,
  `contextWindow: 114688`) + config.yml modelRoles ×14. Runbook renamed:
  `serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-zealot (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
  Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k
  contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb):
  22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's
  ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), decode
  48.0 t/s, vision `Red`. Evening: swap to
  `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`
  RCO-IQ3_XXS-mtp (10,442,827,392 B, sha256 `e13d3e74…`; 1.56 GiB lighter)
  + the publisher's own `mmproj-BF16.gguf`, MTP back on (draft-mtp depth
  3) → commit da36fa3: **23,636 used / 489 MiB free** (the tightest fit in
  the endpoint's history — the draft ctx measured ~2.4 GiB, built against
  the target's full 262,144 context, not the 3-token proposal depth; the
  lighter quant funds it), decode 63.3 t/s recipe / 72.9 temp 0 (acceptance
  49.8–62.4 % vs 48.0 no-spec), vision `Red` at a 128-token budget (the
  uncensored model thinks more — 64 tokens is all `reasoning_content`).
  ngram-mod still skipped (recurrent-state rollback law on the qwen35
  hybrid). Runbook renamed + rewritten:
  `serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-dragoon (Dankclaw): halogen 27B-dense → halogen-flash-server
  0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
  commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
  flash-server line, so the engine moved with the model. Abliterated base
  (refusals gone), vision back (the v2 tower loads beside the checkpoint),
  MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
  Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
  **Incident (03:09:39–03:11:42):** the deploy started the unit while host
  `MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
  requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
  crash-looped 8 boots, each dying in ~5 s at the pin gate, and
  self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
  pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
  left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
  again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks
  `MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
  `FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
  it never does (the 16 GiB floor is the engine's, not
  `HALOGEN_HOST_RESERVE_GIB`).

## 2026-10-05
- **frank-archon: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
  dankserver.** Built FreeToken at PR head `77a7201` on the host into the
  local image `frank-archon-freetoken:pr587-77a7201`
  (digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
  bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
  symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
  `~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
  through git (env/serve/unit + seam + deployer): ONE R9700, experts in
  pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
  299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
  page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
  `reasoning_content`; models.yml flipped accordingly). Container renamed
  to the unit name (Frank's call; `frank-archon-qwen` → legacy list);
  deployer image gate → `podman image exists` (local builds carry no
  RepoDigests) and health gate reads the BODY — FreeToken 200s with
  `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
  (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
  completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
  "Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
  single-stream (vLLM was ~30 on two cards; the KV contract caps the
  single-card LRU; TP=2 exists but the pynccl JIT can't compile for
  gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
  Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
  repointed. Rollback: tag `frank-archon-vllm-shapeG-20261003`.
  Runbook: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
  skills read in full and merged, one polished reference page per
  skill-cluster:
  - **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility,
    Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and
    Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget,
    vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile
    Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis,
    Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split
    Triage, FreeToken Builds, plus deltas folded into the retired runbooks
    (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
    FLM pack registration, ik hybrid envelope, frank-medium split
    rebalance, frank-archon llama.cpp GGUF preflight, batch-ubatch split
    section, bench-window D-state/stale-read notes, second-endpoint
    bring-up in Model Swap).
  - **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
    State Triage.
  - **Operations (6 + index):** Remote Job Runner, Payload and Disk
    Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit
    and Posture, SSH and Remote Access.
  - **Infrastructure (2 + updates):** Observability Provisioning, Homepage
    Status Probes; updates to Authentik (guard-chain flip recipe, `ak
    shell` mechanics, user-provisioning playbook, outpost netns probe),
    Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver
    drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration
    (filesystem/rename caveats), OpenWebUI (introspection, verification
    battery, ComfyUI imagegen), Hermes (config-home drift, live config
    edits, auxiliary routing, password rotation — the pre-existing page was
    preserved and extended).
  - **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version
    split, schema-first probing, auth-posture method, auth-config trap, and
    the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]].
  - **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote
    IPC, palette recolor, offline template render, rotation pinning, iCX3
    RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center
    Dotfiles, index), PICO-8 Asset Standards.
  - Index updated: `LLM/index.md` gained a Cross-Cutting References
    section; Operations/ and Agent/ got index pages. AGENTS.md now points
    at the new sections. Link checker + full test suite green.
- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
  project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
  links that terminate on 192.168.0.112, so every deploy gate died there when
  the Mac mini slept (Caddy 502 on the five vhosts, connection timeout on the
  raw Sonarr link). Split: those links moved to
  `tests/e2e/specs/19-frankbook.spec.ts` under a `frankbook` project;
  `tests/e2e/run.sh` excludes it from the default project set, and
  `scripts/deploy.sh` computes a trigger from the same BASE..HEAD range the
  plan uses (the spec, `frankbook-pro/**`, `shared/homepage/services.json`,
  the `dankclaw/Caddyfile` arr vhosts) and sets `E2E_INCLUDE_FRANKBOOK=1`.
  Verified live with the host asleep: full suite 33/33 in 1.1 min, the admin
  walk back to 5.4 s; `--project=frankbook --list` proves the selection.
  Live pass of the frankbook walk itself is deferred to the next deploy with
  an awake host.
  - **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
    left `frank_zealot_unit.bats` and the deployer's payload gate stale** —
    the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
    `LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
    loads (the anti-drift seam doing its job). Both repointed to the
    Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
- **frank-zealot: compute-buffer pair doubled to 4096/2048, and the prefill win
  measured rather than assumed.** Frank asked for the batch and ubatch doubled.
  The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
  failed the scratch boot"**, and this shape carries 616 MiB less free space
  than that predecessor did.
  - The A/B used the service itself as the bench (one card, one server — no
    scratch port): the old rung was re-booted through a transient
    `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
    boot **byte-identically** (21,276 used / 2,849 free), which is what makes
    ubatch the only variable. Repo never saw a fake revert commit; scaffolding
    staged to `~/.AGENTTRASH/20261005-frank-zealot-ubatch-ab/`.
  - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
    for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
    noise). The control that proves it is geometry and not drift: a 279-token
    prompt sits below *both* micro-batches, so it cannot see the knob, and it
    did not move (2,676 vs 2,658–2,811).
  - **The predecessor's boot failure did not reproduce** — that was a shape
    observation, not a law; this shape boots 2048 clean. And the fleet lesson
    "**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
    true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
    CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
    — the curve is backend-specific and must never be carried across backends.
  - The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
    and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
    asserts the argv follows the shipped env plus the `ubatch <= batch`
    invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
    green, inverting to 2048/4096 turns it red. (The old assertions only ever
    exercised the serve-script *defaults*, because the test env never carried
    these two keys.)
  - `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
    edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
    flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
    second burst and ~90 s). Docs:
    [[frank-zealot]] (*Compute-buffer geometry*).

## 2026-10-03
- **frank-archon: shape F then shape G — MNBT 2048, the CUDA-graph plan term
  pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
  proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
  space; the audit found the arena is `budget - non-expert - KV reserve -
  graphs - overhead` where the only deploy-reachable terms are the budget and
  the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB
  overhead are image constants; the ~3.5 GiB of driver-free VRAM was
  estimator slack, not allocatable).
  - Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices
    overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT
    4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base +
    12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB /
    7,653 slots, budget 31.29.
  - Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
    single weight loaded** — `ValueError: Free memory on device cuda:N
    (31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
    The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
    budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
    the law, never 1.00" — now with a second proof.
  - Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
    0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
    capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
    prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
    3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
  - MEASURED live: arena **17.77 GiB / 7,762 slots** (+240 over shape E),
    KV **401,408 tokens / 2.00x concurrency** — the first shape holding two
    full-length conversations with real spare; capture measured 0.31/0.10
    GiB against the 0.07 charge (under-charged, boots and holds); driver
    used 30.97 of 31.86 GiB -> **0.89 GiB real free per card**, held through
    a live 4,096-token generation (~38-39 tok/s single-request wall-incl.,
    +13% vs the 33.99 shape-E baseline; single sample, not a bench).
  - Verification seam rewritten to assert the shape by its OUTPUTS (util at
    the 0.982 law, graph charge below the analytic default and per-graph
    positive, KV pool >= 400,700 tokens at the measured 7,892 B/token).
  Runbook: `serverAppConfigs-docs/LLM/frank-archon.md`.

## 2026-10-01
- **frank-archon rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
  GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
  served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
  captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
  interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
  draft+hidden groups, registry method-signal), but **acceptance pinned at
  1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
  aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
  8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
  (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
  not originate from this engine (APIServer access log: every completion
  POST was host-local; his workstation appears once, a /v1/models GET).
  Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
  (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
  (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
  KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
  unifier/tailpin/grouping/registry fixes, apply-verified).

## 2026-09-30
- **frank-archon cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
  the 35 tok/s target, and the shortfall is measured rather than inferred.**
  - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
    group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
    113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
    expert offload: its `quant_method: auto-round` is overridden to `inc`, which
    routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
    compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
    182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
    offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
    refused to serve at all. That checkpoint was purged with the other retired
    payloads.
  - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
    cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
    `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
    `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
  - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
    (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
    rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
  - **The wall — and a corrected one.** The first account of the shortfall
    bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
    in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
    hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
    (496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
    kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
    `vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
    per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
    **1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
    GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
    fusions that DO happen 6 %. The real cause is that the fork compiles its own
    Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
    **not** its GLM trunk, so every pointwise op is a separate eager kernel; the
    engine says so at boot ("...provides neither a compiled submodule nor
    breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
    13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
  - **Fork work staged (compile changelist, WIP — does not boot yet).**
    `scripts/patches/glm53-compile-01-model-decorator.patch`,
    `-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
    evidence header and base/patched sha256. Boots established, in order: (1)
    `functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
    wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
    available -> os.path.exists` breaks the graph (a plain module-global read
    compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
    breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
    `fullgraph` because its indexer metadata syncs to host
    (`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
    outright. With all three the trunk **traces** (`Dynamo bytecode transform
    time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
    mHC/KDA path — a latent bug the compile exposes. That is the remaining
    blocker.
  - **No new HIP kernels are possible** (the image ships `r4d.so` with no
    sources), which binary-blocks the second-largest lever: the fused router is
    gated `num_experts in {128, 256, 512}` and only
    `r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
    experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
    reserve is dtype-blind), only removing gfx12's only MLA decode path.
  - **DFlash2 for GLM is a target-side port, not a drafter port.**
    `incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
    and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
    `target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
    (`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
    `"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
    capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
    demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
    proportional to its token count.
  - **DFlash2 tried, and the gap was exactly that one interface — now closed.**
    Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
    2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
    `--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
    accepted the config and the drafter and died at worker init with
    `RuntimeError: Model does not support EAGLE3 interface`. The port
    (`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
    hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
    text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
    wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
    `Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
    `pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
    workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
    DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
    by experiment-window RAM contention (see the window discipline below), so the
    drafter-serving end state was still unproven when this entry was written.
  - **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
    With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
    dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
    reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
    `mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
    indexer's in-forward workspace acquisition
    (`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
    _ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
    — and the cause was **my own scratch launcher**: its compile branches were
    restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
    boots ran `fullgraph=True`, where *every* break is an error and the first one
    wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
    (the flag only logs). The moving reports are still useful — they are the break
    list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
    patch 05 makes the workspace acquisition compiler-opaque.
    `scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
    the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
    sets at `attention.py:583`).
  - **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
    `VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
    hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
    `VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
    near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
    recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
    chaining the image's apport hook), and the scratch launcher used to reset
    `ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
    All four patch artifacts are now genuine `diff -u` patches, verified to apply to
    the pinned image and to reproduce their recorded sha256.
  - Measured and rejected: **MTP4** (the head loads, arch resolves to
    `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
    decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
    The profiler window above explains it: the per-token cost is dominated by
    pointwise glue that scales with the token count, so verifying a draft batch
    pays ~N x that glue for the same N tokens of output);
    **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
    aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
    that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
  - Fork bug found: the plan's KV reserve under-models this model's
    per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
    that reserve, making a legal `--max-model-len` unbootable. Shipped
    `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
    request raise the cap) — NOT needed by the deployed shape.
  - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
    state dirs are intact on disk.


## 2026-09-29
- **frank-dragoon tuning investigation: a measured NEGATIVE result, nothing
  changed.** Frank asked for iterative decode/prefill improvement at a 32k
  working point without touching the model or quantization. **No win exists
  at gufo's served settings**, so no config change was shipped. frank-dragoon
  was stopped for the whole investigation (~4h) per the benchmarking policy
  and restored healthy in 9s afterwards.
  - Measured on the repo's own prose and shell sources at **32,762 prompt
    tokens**, salted per run so a warm prefix cache could not flatter a
    later run, 3–5 repetitions per configuration.
  - **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s.
    A 16x change in chunk size, 0.4% out.
  - **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all
    inside the run-to-run noise band. The 5-rep confirmation round
    **reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came
    **last** (12.86), while the **deployed config came first** (14.71).
  - `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`:
    **rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported
    for Q8 targets** (`Unsupported Qwen MTP tensor type for
    token_embd.weight: Q8_0`).
  - `--context 262144 → 40960` was tested as a diagnostic (the deployed value
    preallocates KV ~8x larger than a 32k conversation needs): **no gain**
    (13.93 vs 14.71 t/s), and it would break `models.yml`'s declared
    `contextWindow: 262144`. Not shipped.
  - **What actually moves the number is speculation itself: 6.65 t/s off →
    ~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to
    everything tried, consistent with gufo's own `EXPERIMENTS.md` where
    dozens of kernel variants are recorded rejected for "no release
    throughput improvement".
  - **Two traps are now in AGENTS.md's benchmarking policy**, because they
    cost most of the wall time and both produced *false* results:
    (1) **`gufo bench` cannot rank speculative settings and inverted the
    ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an
    identical trace sha256 across draft widths (the same highly-predictable
    text every run, 93% of proposals accepted). It reported 29–37 t/s where
    real traffic gives ~15, and made dt16 look 6% faster than dt7 when real
    text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**;
    at 8, `inter_token` is first-token noise and the first pass of the sweep
    produced a nonsense ordering that the 128-token pass reversed.
  - Also worth recording: the `Server-Timing` header is
    `ttft;dur=402.485` — the unit is in the **name** half, so splitting on
    `=` keys the dict `ttft;dur` and every lookup silently returns nothing.
    Four configurations reported "no Server-Timing" while the header was
    present on all of them.
- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s,
  mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4
  fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full
  flag-level A/B battery at the 32k working point; every number is from the
  server's usage counters (SSE chunks are invalid under spec decode). New
  runbook: [[frank-archon]].
  - Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1,
    TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx,
    acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
  - **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64
    (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the
    stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in
    ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves
    the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral);
    prefetch depth 2 (neutral — the tier is 16 slots/layer against a
    ~2,400-slab working set at 73% misses); 2-way concurrency (summed
    25.49 < single 26.8 — under TP both streams' misses gather on each
    rank, the links do not add capacity, and the shared arena thrashes).
  - **TunableOp rejected:** it silently skips under expert offload unless
    `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics —
    repetition loops, the server's repetition-halt firing at ~140 of 256
    tokens, "79.8 tok/s" measured on halted degenerate completions, and a
    fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no
    persistent cache in the container lifecycle.
  - **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free
    at worker init); `--max-num-seqs 2` (compile constraint violation at
    CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not
    prepared" — PLE prep is wired only for the MTP spec path).
  - **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction;
    the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91%
    of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s =
    42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait
    (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the
    fork's gather read path — fork-patch territory, not flags.
  - **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm
    (>10 min, two attempts, while the AsyncLLM wrapper logs success in
    1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min
    profile window's trace buffer ~40 GB → the kernel OOM-killed the engine
    core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer
    expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY
    through that stop path, so the telemetry JSON is unreachable without a
    fork patch.
  - **256k ctx boots** (the last original-goal dimension): a 262,144-token
    window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈
    16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused
    (`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate
    at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77**
    (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just
    under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
    Correction recorded: the planner sizes the arena AFTER the KV override
    (the earlier "before" reading was wrong) — the arena is what remains.
  - **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
    boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
    engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
    4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
    (5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
    Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
    band, straddling the 35 target within noise. Final 256k config:
    `MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
    alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
  - **GLM-5.3-Flash readiness REVISED by source reading (Frank's question
    about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container
    registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split
    module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA);
    `offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts,
    Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with
    the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit`
    default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback.
    **`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` /
    `moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing
    kernels" GLM verdict is dead for this fork. WNA16 (int4) backend
    oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are
    CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16
    group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4
    (~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
    GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
    (> 246 GiB RAM). Feasibility doc rows updated.
  - **Revert pins installed (Frank's request):** (1) repo tag
    `qwen-fp8-tuned-20260929` at f532e03 (the campaign's docs + measured
    config; `git checkout qwen-fp8-tuned-20260929` restores the state);
    (2) image pinned by digest — serve script now launches
    `docker.io/tcclaviger/vllm@sha256:da402966...` (ID 3322dbbbc96d) with a
    local alias `qwen-fp8-runtime:tuned-20260929`; the floating `:latest`
    is no longer referenced anywhere (the other image on the host,
    `28.06.1-Code000` = fffca39f, is a DIFFERENT build — do not confuse).
    Runbook deployment-shape section updated.
  - **Endpoint published:** `PORTMAP=7773:7770` hook added to the serve
    script; the 256k config relaunched with it. Host-side checks: health
    200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
    qwen3.8-flash-next, decode through the published path
    36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-archon
    owns 7770 when it ever returns.

## 2026-09-28
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
  clean up old models. **510 G staged** to
  `~/.AGENTTRASH/20260928-retired-models/` — moved, NOT deleted, with a
  `MANIFEST.md` giving a restore command per item. **Nothing is freed until
  the purge is approved**; that is the point of the trash policy, and the
  number to know before expecting the disk to move.
  - Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G,
    mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G,
    ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter
    838 M. Every one maps to a runtime already retired in the runbook index.
  - **The 29.3 G rollback target went with them**, by Frank's decision. The
    payload-only rollback is now a re-fetch first; both the runbook and
    AGENTS.md say so. **The DFlash2 drafter was NOT reclaimed** — only the
    `.gguf` was taken out of that directory, because the live frank-dragoon
    drafter lives in it. The manifest warns against restoring the directory
    wholesale.
  - **Kept deliberately:** `checkpoints/` 23 G. It is unreferenced by the
    repo, but ComfyUI is live and its workflows were not searched — not
    confirmed dead, so not taken. `logs/x4e-run.log` also stays: it is a
    **sparse** file, 3.3 G apparent but 4 K real, so removing it frees
    nothing.
  - **A live reference check came first, and it earned its keep.**
    `serve-frank-zealot.sh`'s *fallback* `MODEL_DIR` still pointed at the
    retired GSQ-RCO payload frank-zealot served on 2026-09-16/17. The env
    file has always overridden it, so the drift was invisible — but a
    missing or unreadable env file would have turned a config problem into a
    "missing asset" naming a model that no longer exists. Repointed at
    `Gemma4-12B-QAT/`, which is what the env file already sets. Every other
    hit was a comment, a retired fetch script's destination, or
    `retire_halogen_dragoon_quadlet` (which moves a *quadlet*, not a model).
  - **Not taken, by Frank's decision — scope was models only:** podman
    images (**279 G reclaimable, 95%**; 170 images, 23 active) and the
    uv+pip caches (**73 G**). Together that is 352 G of regenerable garbage
    larger than the models reclaimed, and it is the obvious next move.
  - Pre-existing: **118 G of trash from 2026-09-21** was already staged
    (`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never
    purged. Total now staged on this host: **627 G**.
  - Verified after the moves: frank-dragoon and frank-zealot both `active`,
    both `/health` 200, and all three live assets (Swift target, DFlash2
    drafter, Gemma4 target) still present.
- **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production
  change, Dankclaw). Frank asked whether the UD Q8 of
  `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
  gufo, then said to do it. Port 7770, alias `frank-dragoon`, one 262,144
  session, the DFlash2 drafter and the client contract are unchanged; the
  27B dense weights are an **abliterated (uncensored)** fine-tune of the
  same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
  - **Compatibility was established from the GGUF headers, not from the
    filename.** Both files were range-fetched and parsed: same
    `general.architecture` (`qwen35`), same 866 tensors with **identical
    names in identical order**, **zero** per-tensor `ggml_type` or dim
    mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
    geometry. The two scary metadata differences are both no-ops, confirmed
    against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
    not consulted"* by `gguf_reader.cpp` — the file gufo already served
    records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
    `qwen35.attention.recurrent_layers` key is **never read** by gufo
    anywhere in `src/`; layer kinds come from `full_attention_interval`,
    which both files carry as `4` and which agrees with it.
  - **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
    matching the Hub LFS oid, via the new
    `/scripts/download-frank-dragoon-swift.sh`. It sends **no auth header** —
    the repo is public and ungated, and a token on the curl command line is
    world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
    ~1.8 MiB/s; resuming with the host's token through a **mode-600
    `curl -K` config** took it to ~7 MiB/s. The download survived an ssh
    drop by running `setsid nohup`, not under the session.
  - **Load test, service stopped first** (the GPU policy): **9.2 s** from
    `load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
    context capacity 262,144, no loader warnings. Contract battery green:
    `context_length 262144`, `content` + `reasoning_content`,
    `finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
    reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
    lockpicking scene it wrote one without hedging, confirming the
    abliteration.
  - **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
    template"; gufo **cannot** take it. This is not a missing flag —
    `src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
    that appends strings, with no Jinja engine, no `--chat-template-file`,
    no env override and no template body in the binary. Qwen-Sharp
    v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
    `raise_exception`, so supplying it is a **port, not a config line**.
    Frank chose to ship on gufo's stock renderer and accept the loss: the
    `terse` system-prompt injection, in-band `<|think_*|>` control, the
    `json|xml` `tool_call_format` switch, tool-argument truncation and
    `add_vision_id` prefixes are all gone. gufo's renderer does keep the
    effort instruction strings and `preserve_thinking`. **Worth watching:
    an abliterated model now runs without the prompt that tells it to stop
    waffling** — the cheap fix is to send the `terse` text as an ordinary
    system message from `models.yml`, not to touch a template.
  - **The payload is now two files in two directories.** The target moved
    repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
    tunable. That broke the deployer's stray-run sweep, which keyed on the
    *target* filename and would have silently stopped matching — a stray
    run would then have held 7770 past the port gate. It now keys on the
    drafter, which every frank-dragoon command line carries via
    `--dflash-model` and which did not move.
  - **`/tests/frank_dragoon_unit.bats`** — 20 assertions. The mutation pass
    found **four real gaps** on the first run, all now fixed: two fetch
    assertions were satisfied by a **header comment** rather than the URL
    assignment, so repointing a repo left them green; and nothing bound the
    env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
    actually download, so a re-upload or an env edit would break the
    endpoint with a "missing asset" no test predicted. There is now an
    explicit cross-file invariant. A second harness bug worth recording:
    a `bats --filter` that matches no test reports zero `not ok`, which
    looks exactly like a mutation that survived — the harness now asserts
    the `1..N` count before believing a result. 16/16 mutations caught.
  - Docs: runbook gained a **Payload** section (the header evidence),
    **The template law** section and a two-way **Rollback**; AGENTS.md
    fleet table, frank-dragoon section and rollback.
- **frank-dragoon runtime swapped to gufo** (production change, Dankclaw).
  Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
  built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
  `gfx1151`) — in place of the stock llama.cpp prebuilt. Port 7770, alias
  `frank-dragoon`, one 262,144-token session and the client contract were all
  unchanged; only the engine was new, and the payload was the unsloth
  Qwen3.8-27B UD-Q8_K_XL **at that point** — it was swapped for the
  abliterated Swift-1.5 later the same day, see the entry above.
  - **The runtime is built from source on the host.** gufo ships no prebuilt
    Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host
    has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to
    `/home/dank/GitPackages/gufo-v0.1.1/` by the new
    `/scripts/build-gufo.sh`, which is reproducible and fails closed on a
    moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host
    ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay.
  - **Two host workarounds, both encoded in the build script.** Ubuntu's
    `librocwmma-dev` ships a truncated rocWMMA header set into
    `/usr/include` that shadows AMD's complete one; the build died on
    `'internal/accessors.hpp' file not found`, and a `-I` cannot fix it
    because in HIP mode the host compilation pass searches the system
    directories first. The package has no reverse dependencies; it is
    removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm
    clang prefers the newest tree, whose `<atomic_wait.h>` needs a
    `std::__detail::__notify_impl` symbol version the installed
    `libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until
    `--gcc-install-dir` was pinned to GCC 15.
  - `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu
    loaded, 20 CUs). It self-caps compute queues at 2 resident for
    `serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`,
    which matters because ComfyUI holds queues on the same device.
  - **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive**
    block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is
    gone), greedy, `--think auto --preserve-thinking on`, text-only.
  - **Client-contract deltas, all in the runbook:** the chat template is
    compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp
    73k-token-prompt failure mode cannot recur); `reasoning_effort` now
    accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold
    is no longer required — **kept anyway** so this commit does not also move
    every `high` role from `xhigh` to `high` effort; and there is no
    KV-quantization or prompt-cache-RAM flag.
  - **Measured at cutover:** 13 s from `load_started` to listening (target
    8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
    RSS 30.6 GiB; a live short-prompt request returned
    `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
    Verified live: `/health` 200, `/ready` 200, `/v1/models` →
    `frank-dragoon` at `context_length 262144`, a chat completion carrying
    `reasoning_content`, a `tool_calls` response with
    `finish_reason: tool_calls`, and SSE streaming.
  - **The trade, stated plainly.** gufo's own Q8_K_XL tables against
    llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
    +7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
    131,072**. On the 8-bit target the crossover is ~32k and frank-dragoon runs
    one 262,144-token conversation, so long agent sessions sit on the losing
    side of it. The Q4_K_XL target shows no such regression; revisiting the
    quantization is a separate decision.
  - `/tests/frank_dragoon_unit.bats` — Rewritten around behaviour: the seam
    runs the serve script against a stub engine and asserts on the argv it
    actually execs, instead of grepping the script's text. A caveat found
    while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
    a test** (bash exempts the non-final element of an AND-list from
    `errexit`), so every such assertion was split into standalone `[ ]` lines.
    Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
    a wrong context/sessions/port, an alias swap, a moved drafter, a removed
    missing-asset message and a neutered commit pin each fail it.
  - `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` gates on the runtime
    binary as well as the payload and returns **before** it stops the serving
    unit, so a host whose gufo was never built reports the failure instead of
    taking the endpoint down on the way there.
  - Docs: new `/serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md`;
    `frank-dragoon-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
    rollback target — the prebuilt and the payload are both intact on disk);
    AGENTS.md fleet table, frank-dragoon section, bench policy and runbook
    index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
  - **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
    Mac mini) at 100 % packet loss — offline, and the same failure recorded
    in `564a781`. Not related to this change; no frank-dragoon test failed.
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
  production change). Frank asked which of DeepSeek-V4.1-Flash,
  MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
  and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
  them fast. Findings, all evidenced:
  - **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
    of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
    240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
    stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
    **~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
    dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
    layer-major prefill for V4.1, so the only legal prefill is token-by-token.
  - **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
    claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
    gfx1201, zero errors). The 110 works because the 101 GiB container leaves
    the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
    quantiser, container, slab, host pool and placement engine (expert tensor
    naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
    new indexer, the DSpark draft geometry and Engram — and Engram fails
    *silently*, because V4.1 has no `num_hash_layers` key so Affinity's
    `model.h:36` default of 3 survives. Weeks of work, same capacity wall.
  - **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
    `feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
    R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
    246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
    downloading for a three-arm bench.
 - `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
   60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
   runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
   (244 G) and must not be planned for payload.
 - `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
   runbook; indexed in `LLM/index.md`.
 - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
   mid-session to free space. frank-archon is still answering `:7770` from
   mmapped-but-unlinked inodes (`/models` is empty inside the running
   container) — **the endpoint cannot be restarted until the 175 GiB payload
   is re-fetched.** A reboot or a container restart takes frank-archon down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
  (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
  MERGED to main + pushed AND already live on frank-archon: the running
  `frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
  `--enable-prefix-caching` in argv and `VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800` in env,
  `/health` 200 — but `deploy-state/frank-archon` was still stamped `f187165`. A
  `deploy.sh` run would have re-selected frank-archon and reloaded the 175 GiB
  payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself
  runs on. Stamped the marker `f187165 → ba2100e` (status `deployed`, atomic
  mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone
  failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
  (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
  regression.
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
  frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
  VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
  completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
  in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
  and reports nothing); probe via the shim's `docker` or the unit MainPID.
- `/scripts/frank-archon.env`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-archon prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
- **Incident — frank-archon down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.

## 2026-09-27
- `/scripts/frank-archon.env`, `/scripts/frank-archon.service`, `/scripts/serve-frank-archon.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_archon_unit.bats`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-archon cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-dragoon. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
- Dankserver frank-archon slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
- Dankserver frank-archon restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-archon-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-archon.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-archon.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.

## 2026-09-25
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-dragoon → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-zealot → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-dragoon = deep reasoning, frank-medium = primary model, frank-zealot = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-dragoon listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).

## 2026-09-22
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-zealot-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-dragoon (unsloth UD-Q8_K_XL) and frank-zealot (Gemma4-12B).
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.

## 2026-09-21
- `/dankserver/frank-medium/quadlet/frank-medium.container` — frank-medium cut back to the MXFP4 checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`); `KV_DTYPE` became a quadlet tunable, set `auto` (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
- `/dankserver/calibre-server/quadlet/calibre-server.container` — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, `--auth-mode=basic` pinned, internal port 8090.
- `/serverAppConfigs-docs/Dankserver/Calibre.md` — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
- `/scripts/serve-frank-dragoon.sh`, `/scripts/frank-dragoon.env`, `/scripts/frank-dragoon.service`, `/scripts/download-frank-dragoon-unsloth.sh` — frank-dragoon (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
- `/serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md` — New runbook; `frank-dragoon-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
- `/shared/prometheus/prometheus.yml` — `node-dankclaw` target fixed to `host.containers.internal:9100` (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); `Infrastructure/Prometheus.md` targets table + `tests/observability_quadlet.bats` contract updated.

## 2026-09-20
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-zealot/frank-dragoon targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.

## 2026-09-19
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
- `/serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md` — New runbook.
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.

## 2026-09-18
- `/scripts/frank-medium.env` — `CACHE_RAM` 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; `tests/frank_medium_unit.bats` seam added; `--ctx-checkpoints 0` confirmed a deliberate deviation (build default is 32) (1acca43).
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.

## 2026-09-17
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
- `/scripts/frank-zealot.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
- `/serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md` — New ACTIVE runbook; `frank-zealot-qwen38-gsq-rco.md` retired as rollback target.
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.

## 2026-09-16
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — halogen restored as frank-dragoon, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-zealot); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
- `/scripts/frank-zealot.env`, `/scripts/frank-zealot.service`, `/scripts/serve-frank-zealot.sh` — New frank-zealot: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
- `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
- `/serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-dragoon-ik-uncensored.md` marked retired.
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.

## 2026-09-15
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh`, `/scripts/frank-medium.service` — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (`retire_root_llm_small()` archived the CUDA-era unit).
- `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` — New runbook; `freetoken-frank-small.md` and `kingjones-q8agent-rocmfpx.md` marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
- Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the `ld.so.conf` hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`).
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
- `/serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md` — New runbook with the full evidence chain.

## 2026-09-14
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — New `hermes-webui` service (Hermex): `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data`, loopback-only publish 127.0.0.1:8787, password-gated via `HERMES_WEBUI_PASSWORD`.
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories.
- `/tests/hermes_webui.bats` — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
- `/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md` — New page; remote access = Termius local forward over the existing router :22 port forward.

## 2026-09-12
- `/dankclaw/marreta/quadlet/` — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; `NGINX_HTTP_PORT=8085` (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
- `/tests/marreta_quadlet.bats` — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
- `/serverAppConfigs-docs/Dankclaw/Marreta.md` — New page (incl. the `base64:`-prefixed APP_KEY gotcha).
- Live — authentik ProxyProvider `marreta` (pk 13) + Application link: the outpost's config API filters `application__isnull=False`, so a provider with no linked Application is never loaded (the login-wall root cause); a null `invalidation_flow` crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec `16-marreta.spec.ts`.

## 2026-09-04
- `/frankbook-pro/` — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, `/opt` config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed `10.88.0.1:8080` → `127.0.0.1:8080`.
- `/tests/frankbook_quadlet.bats` — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
- `/dankserver/` — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned `/opt/<svc>/config` dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and `userns_mode` must NEVER be added (podman-compose runs services in pods; `--userns` + `--pod` is rejected).
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
- `/tests/e2e/` — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).

## 2026-09-03
- `/serverAppConfigs-docs/` — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, `Prowlarr.md`/`prowlarr.md` deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.

## 2026-09-01
- `/frankbook-pro/gluetun/`, `/frankbook-pro/qbittorrent/` — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (`WebUI\Password_PBKDF2`, not the legacy `WebUI\Password` key qB ≥ 4.2 ignores); `DefaultSavePath` → `/Volumes/COMPLETE/Finished Torrents`; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
- `/scripts/deploy-remote.sh` — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + `HostHeaderValidation=false` + the four categories; full 7-service deploy run.
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — qB pinned `4.6.7` (5.x's `204` login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
- `/serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md`, `/serverAppConfigs-docs/Architecture/Storage Policy.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.

## 2026-08-30
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — frank-dragoon: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
- `/scripts/frank-zealot.env` — frank-zealot (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).

## 2026-08-29
- `/dankclaw/Caddyfile` — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
- `/dankclaw/anubis/policy.json` — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a `--service anubis` deploy restart is mandatory after policy edits.
- `/tests/authentik.bats` — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile).
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Infrastructure/Anubis.md`, `/serverAppConfigs-docs/Architecture/Network Architecture.md` — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original `Host` vs each provider's `external_host` (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch; the public zone gained the `auth` A record via the Porkbun API (ee1e038).

## 2026-08-26
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.

## 2026-10-01 — frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)

bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
--expert-precise-memory:

- with precise: NRestarts 1->3, each attempt ~6.7 min then
  `Deactivated successfully` (API server drains after an EngineCore fatal
  error) — cause not captured, the serve script rm'd the container.
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
  climbing, container `Exited (1)`, and the journal finally showed the real
  error:
  `RuntimeError: Worker failed with error 'Constraints violated
   (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
   as dynamic but your code specialized it to be a constant (3)'`
  — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
  `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
  fresh container filesystem, so a stale compile cache is ruled out: the
  conflict is produced by this configuration's own compile path.

Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
`--enforce-eager` is the bypass that would sidestep the constraint path.

Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).

## 2026-10-01 — frank-archon: --expert-precise-memory + bf16 = boot crash loop

**Reverted.** The two changes Frank asked for were applied together and the
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
new `Restart=always` turned a silent death into a visible loop).

The budget arithmetic explains it without needing the error text, because the
arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
that is exactly what `rocm-smi` measured):

| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
|---|---|---|
| non-expert weights | 6.28 | 6.28 |
| pinned pool | 6.80 | 5.68 |
| CUDA graphs | 0.55 | 0.55 |
| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
| non-torch | ~1.45 | ~1.45 |
| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
| free at boot | **~0.13 GiB** | 3.59 |

`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
put graph capture ~0.13 GiB from the ceiling, where every transient allocation
dies. So the flag is dropped and bf16 2 x 262144 ships with the default
overhead: arena 14.31 GiB, ~2.4 GiB free at boot.

**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
default — it is the budget for the phase that allocates last. Reclaim it only
by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).

## 2026-10-01 — frank-archon: 2 bf16 windows + --expert-precise-memory

Frank's final shape for the night. Two changes at once, both measured in
principle before the boot:

- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
  bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
  11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
  1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
  524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
  3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
- **`--expert-precise-memory`.** The registry's default overhead for this
  architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
  nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
  The flag selects the fitted figure, which moves only the plan's `room` term
  — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
  cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
  1024 nbt above 4096, so MNBT 2048 pays nothing.

Budget note worth keeping: the arena is `(budget - non_expert_weights -
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
planned arena; it comes out of free VRAM instead.

## 2026-10-01 — frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions

Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
uncensored). New fetcher `scripts/download-frank-dragoon-q4.sh`; the DFlash2
drafter and its fetcher are unchanged.

Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
to `kFp16`), so "unquantized" is inherent.

KV arithmetic (why this fits where one 262144 session used to sit):
f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
~62 GiB against the 114 GiB GTT.

## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner

`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
deploy gate twice and then passed cleanly on a manual re-run:
- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
  — Plex redirects its web UI to the hosted app; the same failure appeared on
  the previous deploy too.
- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
  — the homepage answered `http=200` in 40 ms from the dev box at the same
  moment, so this is a browser-level flake.
- run 3 (manual): **33 passed**, exit 0.

So a red gate on this spec alone is NOT proof the fleet is broken, and the
deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
stable underlying issue and it lives in Plex's web-client/image config, not in
any repo service.

## 2026-10-01 — frank-archon bf16 no-spec cutover

- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the
  night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
  KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
  FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner:
  "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
  request: 3.03x".
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
  (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
  allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
  (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
  PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
  model's native 262,144 does not fit with a 3-seq contract. Every rung was
  booted under a transient systemd unit at uid dank with
  `LimitMEMLOCK=infinity` (the rootless bench probe masked the production
  memory environment) with full engine-log capture.
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
  `--gpu-memory-utilization` shrinks the engine's own budget rather than
  buying slack; Frank forbade it and the winning ladder ran at 0.98.
- `/scripts/serve-frank-archon.sh` — `--speculative-config` line removed;
  `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
  container-env passthrough added; shape header rewritten.
- `/tests/frank_archon_unit.bats` — seam re-pinned: bf16-KV-present,
  spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
  qualified-shape and flags tests flipped to the new env keys.
- Overnight redo campaign (prefetch OFF everywhere) tabled in
  `/serverAppConfigs-docs/LLM/frank-archon.md`
  with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
  1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
  six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
  KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
- `/AGENTS.md` — fleet table frank-archon row rewritten to the bf16 shape.
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
  updated out of the GLM era.
- **Client contract fixed (thinking + tools).** The live endpoint returned a
  null `reasoning` field and dumped the thinking text INLINE in `content`
  with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
  — the malformed thinking block clients choke on — because the deployed argv
  had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
  three contract flags and the qwen rollback inherited the gap. Restored from
  `ecaad36`'s values, confirmed against this fork's registries:
  `--reasoning-parser qwen3`, `--enable-auto-tool-choice
  --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
  '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
  "NO tool dialect" test is flipped to a PRESENT test.
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
  fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
  free) — six full-length contexts do not fit at any KV dtype — so the
  contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
  rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
  5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
  three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
  same night) is the documented max at 3 contexts.
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
  with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
  <= 262,144) and
  `preserve_thinking` moved from a bare top-level body field to
  `chat_template_kwargs` (the form vLLM actually reads).
- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
  passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
  at runtime; a real client request (995-token prompt, `max_tokens=261149`,
  sampling params straight from models.yml) killed Worker_TP1 with
  `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
  the engine shut down. Boot is not proof of runtime survivability — after a
  knife-edge shape change, send a large-`max_tokens` request before calling it
  done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
  GiB per rank for activations/graph capture. Frank's call 2026-10-01.
- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus
  `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
  makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
  "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
  down until a human noticed. An explicit `systemctl stop` still wins, so
  bench windows and deploys are unaffected.
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
  MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
  contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
  and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
  the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
  decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
  arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
- **DCP investigated and closed** (see the parallelism survey in the runbook):
  `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
  TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
  knob on this box.
- Production bring-up: the shape boots and serves on :7770 — engine log
  verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
  contract probes (reasoning split, tool_calls). Rollback target for the
  2026-09-29 fp8+MTP4 shape: git tag `frank-archon-fp8-mtp-20261001`.

## 2026-10-01 — frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP

- **Question that started it:** can a Qwen3.5 MTP drafter attach to
  frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
  not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint
  and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
  digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
  TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
  vLLM reads the draft from a SEPARATE path — so cross-model MTP from
  `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
  geometry) works. Frank's call: ship that as frank-dragoon.
- **Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token
  prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
  is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
  +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
  19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
  arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
  text (bf16 batched verification reorders reductions; near-tie argmaxes
  cascade). vLLM spec decode is distribution-preserving, not
  bitwise-greedy-preserving.
- **Client-contract delta wired:** vLLM emits `reasoning`, not
  `reasoning_content` - `models.yml` frank-dragoon block now sets
  `reasoningContentField: "reasoning"`, and the extraBody moved to the
  Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
  (a top-level `preserve_thinking` is ignored by vLLM).
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
  `model.safetensors` (the dflash layout) and would have rejected the
  deployed MTP arm (4 shards behind an index); fixed to accept both
  layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
  unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
  0.30 (verified in-image); the drafter stays on disk as the alternative.
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
  DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
  rollback pinned at git tag `frank-dragoon-gufo-q4kxl-20261001`; engine,
  Q4_K_XL payload and DFlash2 drafter all remain on the host.
- First deploy attempt #2 crash-looped the container 51 times: the serve
  script handed vLLM the HOST model path while the tree is mounted at
  `/models` inside the container — transformers rejected it as a malformed
  repo id. Fixed by converting to the container path (the spec draft was
  already converted; the target was not), pinned by bats. Also fixed the
  deployer's own draft gate (same single-file bug as the serve script's).
- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
  `scripts/download-frank-dragoon-swift.sh`) and the orphaned flux-2-klein
  tree under `models/checkpoints` (23 G; ComfyUI mounts
  `comfyui-data/models`, not this) purged via
  `~/.AGENTTRASH/20261001-frank-dragoon-reclaim/` — 52 G freed. KEPT the
  1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
  drafter, not a remnant).

## 2026-10-02 — frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP

- 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT`
  payload + MTP from the repo's own bundled head, one session at the native
  262,144. See [[frank-dragoon]].
- The runtime is **forced by the file**: tensor type 103
  (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115) is unreadable by stock llama.cpp,
  gufo and every other loader on the fleet — only the kingjones30/ROCmFPX
  fork can read it, so the engine is pinned to commit
  `36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
  `scripts/build-rocmfpx.sh`. The AGENT routing variant was chosen over the
  card's plain Q8_0 because it keeps 187 output-head tensors at plain Q8_0
  instead of 2, which measured better MTP acceptance. MTP was the whole
  performance story of the shape (roughly +120 % decode, ladder on the
  runbook).
- Rollback: git tag `frank-dragoon-gufo-q4kxl-20261001`; the retired
  vLLM-era fetchers moved to `old/`. Runbook [[frank-dragoon]].

## 2026-10-02 — frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots

- 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision
  in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K +
  q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]].
- **The pool came from two boots, not a formula.** Differencing the boots
  prices the non-KV floor — and prices the vision tower at **~1.4 GiB for
  a 918 MB file**: the encoder's own buffers ride with it. The first 3-slot
  deploy booted healthy but left less than the ordered 2–3 GiB band free,
  which is why the seam now prices non-KV at the measured value and
  REJECTS that shape.
- Three contexts cost window, and that is the tradeoff Frank was told:
  180,224 tokens per conversation, not the native 262,144; a longer prompt
  is a hard `400 exceed_context_size_error`, so `models.yml` follows the
  slot, not the card.
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
  top-level `images` field — that request answered "Brown." for the same
  red image, i.e. it generated from text alone. Only the `image_url`
  content part reaches the model. `/completion` accepts `images` but this
  build still requires `messages`.
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
  mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=`
  (explicitly empty) in the env file is the documented text-only opt-out
  and returns the tower's ~1.4 GiB to KV.
- Reclaim staged, **not purged, awaiting approval**:
  `~/.AGENTTRASH/20261002-retired-llm-payloads/` of superseded payloads.
  Runbook [[frank-zealot]].

## 2026-10-02 — frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts

Frank's order: shrink frank-archon to a 200704-token window with about two
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
went live as `e460bbb`, and the shape was then re-applied one law at a time on
top of the proven-good base, root cause first.

- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never
  OOM, never MNBT, never the KV dtype.** Four single-variable boot probes
  against the then-deployed shape (production stopped for each, candidate
  booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with
  `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
  Constraints violated (L['query_start_loc'].size()[0]) ... specialized it
  to be a constant (3)`, raised from AOT `build_guards(strict_error=True)`
  during `determine_available_memory -> profile_run -> _dummy_run`, ~7 min
  in; SEQS 3 is healthy at both rungs. `model_runner.py` sizes that input
  buffer `max_num_reqs + 1` and `_dummy_run` sets `num_reqs =
  min(max_num_batched_tokens, max_num_reqs)`, so its width is **SEQS + 1**
  for ANY `MNBT >= SEQS` — 3 at SEQS 2, exactly the constant in the
  message; MNBT only moves the compile RANGE, which is why the identical
  error appears at both rungs. `--enforce-eager` bypasses the path but
  drops CUDA graphs. The 2026-10-01 bf16 2 x 262144 failure was this same
  error class; its message named "KV dtype and SEQS" together and nobody
  bisected, so the dtype was blamed — it was a red herring.
- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.**
  The planner sizes expert cache as `budget - non-expert - KV reserve -
  graphs - overhead`, and the KV reserve is registry-priced at **9190
  B/token x max_model_len**, independent of the pool override: d(reserve)
  equalled d(arena) exactly, and the multi-GB pool cut bought **zero**
  arena slots. The previous session's premise — "the pool cut becomes
  arena, ~7,500 slots" — was wrong; the pool cut is a runtime-HEADROOM
  lever (the shape it replaced died with `0 bytes free` on a real
  995-token request).
- **Final shape (option D):** the argv keeps the only slot count that
  compiles; the POOL caps resident conversations (ledger on
  [[frank-archon]]). Measured live, one prediction corrected: the arena
  prediction landed exactly, the concurrency one did not — **the per-token
  KV price is shape-dependent** (7,701 B/token at MAXLEN 262144 vs **7,892**
  at 200704, ~2.5 % dearer). Never extrapolate a concurrency figure across
  geometries. Smoke probes confirmed the 3 slots real, native `tool_calls`,
  and that `reasoning_effort` accepts only {xhigh, medium, low}, answering
  400 for anything else.
- **Harness contract followed the window:** `models.yml` `contextWindow`
  262144 -> **200704**; side effect: frank-archon's compaction request fits
  inside a frank-zealot slot again. A comment-only follow-up `295d578` was
  deployed so the per-service revision marker sits at HEAD. Rollback for
  the pre-shape-D fp8+MTP4 era stays git tag
  `frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]].

## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service

Found by accident, and it had been latent since the secure-connections posture
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
connection arrives on the container's eth0 - so Plex accepts the TCP connection
and then answers with nothing, and curl reports "Empty reply from server" (52).

- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains
  the whole pending backlog, not just the service you were thinking about.
  plex's per-service revision had been behind long enough that this gate
  had never run against the current posture; it burned its full 600 s and
  failed the run, and every service after plex in the order was left
  undeployed.
- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
  returned 200 with the MediaContainer XML in 44 ms while the gate was
  still timing out; the in-container plain-HTTP path also answers 200 —
  loopback is the one path PMS serves in the clear, which is exactly why
  the host-side probe could not see the difference.
- **The repo already knew this law, in a different file:**
  `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the
  comment "PMS drops plain HTTP off-loopback". The deployer's gate never
  got the same treatment. Fixed to `curl -fsSk --max-time 3
  https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed
  cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e
  33/33, frank-archon untouched (`health=200`, NRestarts=0).
- **Operating lesson: `deploy.sh <target>` with no `--service` is a
  fleet-wide drain.** It restarts every service whose recorded revision is
  behind HEAD and whose paths changed since. To touch one service, name it.

## 2026-10-02 — frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer

Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
of compile headroom the boot never used — and the answer to Frank's free-VRAM
question arrived together with a boot-cost bug that had been costing 181 s on
every restart since the cutover.

- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
  the shape-D base with production stopped, the candidate booted AS the service
  (`FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh`,
  trap-guaranteed restore), the numbers read off the engine's own plan lines:

  | plan line | shape D | shape E |
  |---|---|---|
  | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
  | `expert cache space` | 14.87 GiB | **17.22 GiB** |
  | slots | 6,496 | **7,522** (+1,026, +16 %) |
  | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
  | boot | — | **healthy in 420 s**, no OOM |

  The flag also prints `'expert_precise_memory': True` in `non-default args`, so
  a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
  the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
  flag was the last stack over the line, not the cause.
- **The pool was NOT cut (option A) because the pool is not an arena lever.**
  A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
  1.38x` — measured, rejected as the shipped value, kept as the conservative
  fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
  gains the arena, which is the shape D follow-up had actually been looking for.
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
  Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
  directory`); after our boot it holds `/root/.cache/vllm` at
  **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
  the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
  discards them. The engine names the loss itself: `Compiling a graph for compile
  range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
  `init engine (profile, create kv cache, warmup model) took 217.26 s
  (compilation: 181.20 s)`. The serve script now mounts
  `FRANK_ARCHON_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
  AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
  the comgr JIT cache both persist, and nothing is hidden because the image has
  no such directory. Same class of fix as the recipe's
  `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
  cache key, so an MNBT change costs one recompilation, not a permanent tax.
- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
  direction. Verified a real knob rather than a no-op — the literal is read from
  `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
  alongside the 32x64 gather geometry. **Not A/B measured.**
- **P2P: the hardware supports it, and the engine is already using it.** The
  live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
  on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
  P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
  Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
  the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
  swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
  `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
  - *Acted on:* persistent compile caches (above — the one real operational
    gap), and `GPU_MAX_HW_QUEUES`.
  - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
    list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
    graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
    lighter rank). No 15-52-size ladder to cut.
  - *Not acted on — measured candidates, none shipped (no bench window; these
    are CPU-side and our decode is gather-bound, so a spin-wait stealing cores
    from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8`
    (defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`,
    `HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`,
    `HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals
    are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are
    knobs, not no-ops; any of them needs its own A/B before shipping.
  - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
    fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
    `--override-generation-config` (server-side sampling defaults), published
    port + Caddy rather than `network_mode: host`.
  - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
    `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
    (V1 default), `plecache` volume (the flag is not even passed), and
    `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
    equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
    the 7700X iGPU at GPU[2], so our indices are the right two cards.
- **Harness note, because it cost two probe attempts:** a probe that reported an
  instantaneous candidate death was a logging artifact of the inline SSH script,
  not a crash — the container and its serve script must be polled by recorded id
  and their logs dumped on the exit path, or a failure looks like a bad flag.
  The flag was fine both times.
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
  tokens / 1.99x. Docs: [[frank-archon]].
- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
  `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
  space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV
  cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The
  precise overhead and the arena growth are now production numbers, not a
  probe's projection.
- **The compile-cache fix was then proven on the same endpoint, two consecutive
  boots of the deployed shape:** cold `Compiling a graph for compile range
  (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
  the compiled graph(s) for compile range (1, 1024) from the cache, took
  2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
  create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host
  dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is
  dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s),
  so the honest claim is ~184 s of inductor/AOT build recovered per restart —
  not a proportional boot cut.

## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000

- 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on
  exact attention state, with pool, slab and host pool rebalanced around
  the bigger cache. See [[frank-archon]].
- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the
  whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot
  runs) has **no radiance equivalent**.
- `--max-num-seqs 3 → 2` booted clean: the fleet's "seqs 2 cannot boot" law
  is the retired tcclaviger vLLM fork's torch.compile `query_start_loc`
  constraint, not radiance's — radiance booted seqs 2 in its fp8-native era
  and the sweep had 2 and 3 tied inside noise.
- Harness law re-learned: radiance puts the thinking block in
  `reasoning_content`, so a probe that reads `content` on a small budget
  sees empty text with `finish_reason: stop` — a healthy server scored 0/2
  until the probe set `chat_template_kwargs.enable_thinking=false`. Tool
  committed as `scripts/probe-frank-archon-kv.py`.
- The fp8-vs-bf16 control arms and the open keep-bf16-or-take-cheap-fp8
  call are recorded on [[frank-archon]]; the 2026-10-13 section below (in
  file order above) closes that call in favour of bf16.

## 2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only)

- 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only
  — `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's
  published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other
  argument changes**, mindful the operator session runs on the endpoint
  being swapped. See [[frank-archon]].
- Disk: Frank authorized one `rm` — the spent BF16 source and the
  two-era-old fp8-native `.rad` left `/home/dank/models`; fp8-native is now
  rebuild-only (fetch + build scripts); rollback for THIS swap is the
  previous `.rad`, on disk, by reverting the commit.
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes +
  Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover
  **self-severed by design** — a detached `deploy.sh` survived the session
  death when frank-archon restarted; engine healthy in ~100 s, and the
  prefix tier RECOVERED across the swap (arch+tokenizer unchanged).
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw
  rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`,
  `marreta`, and `marreta-browser` (jobs consumed while gluetun's start job
  retried; `Restart=always` never armed because the services never ran).
  Cascade restarted; skill codified
  (`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate
  passed every deploy.

## 2026-10-15 — authentik: two more human accounts (Dankclaw)

- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM,
  no repo objects**, each `<user>@franksdank.cloud` in the human-user
  template shape (password via `set_password` on stdin only — never argv,
  so nothing lands in `ps`), verified through the exact PasswordStage code
  path; zero `proxysession` rows for either, so no claim purge was due.
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user census
  extended with pk 22–23.

## 2026-10-17 — frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)

- 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two
  slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by
  `scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256,
  `--verify-only`). See [[frank-zealot]].
- Runtime door: the file is `file_type 141` with 402 tensors of custom GGML
  type **142** plus `prism.hadamard.*` metadata — stock llama.cpp refuses
  it, so the PrismML vendor build is **forced, not chosen**; the vendor
  build serves q8_0 V under flash attention, so the source rebuild with
  `GGML_CUDA_FA_ALL_QUANTS=ON` was cancelled as unnecessary.
- Trial-window law cost one boot: the scratch env invented
  `KV_CACHE_TYPE` where the serve script reads
  `CACHE_TYPE_K`/`CACHE_TYPE_V`, so the script's `:=f16` default silently
  won and every arm booted f16. A silently-defaulted knob turns a whole
  trial window into a different experiment.
- New fleet laws recorded (AGENTS.md + runbook): the mmproj's worst-case
  VRAM is NOT inside the engine's fit projection; the first request after a
  boot is a warmup artifact; a 64-token thinking budget makes a thinking
  model look blind; the Bonsai-2 template reads `reasoning_effort`
  (xhigh/medium/low, raises otherwise) so the harness effort fold is
  load-bearing on this endpoint, and it reads `preserve_thinking`.
  Recurrent DeltaNet state bills ~150 MiB **per slot** — pool arithmetic
  alone misses it.
- Base alignment: **Frank chose the OFFICIAL censored payload** — publisher
  provenance over refusal-freedom. The 9B it replaced was uncensored, so
  the homepage card and its warning contract were re-keyed: zealot's card
  now carries the base-alignment notice, the shared legal warning belongs
  to dragoon alone. The CRACK re-export (byte-identical geometry per header
  census) stays on disk as the one-line `MODEL_FILE` fallback if refusals
  bite.
- Two live-path stragglers the cutover had to catch:
  `serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths
  (a defaulted run would have served the old model — now re-keyed to the
  shipped trio), and the OpenWebUI `model` row's `meta.description` is live
  DB state (re-keyed with a DB backup; no `open-webui` restart forced —
  forcing it logs every family session out for one cosmetic string).

## 2026-10-17 — dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)

- Ask: "dump the swift rollback and the GLM GGUF — make sure the model
  caches and agenttrash are empty as well." Preflight first: `lsof` empty
  on both payloads, the serving trio names only the live path, and a podman
  layer census proved no deleted path backs an image container (the 121 GB
  layer is the ENGINE image, not a model blob). Staged, then purged with
  Frank's approval: the two retired payloads, derived kvcache tiers, stale
  compile caches, and every `.AGENTTRASH` on the host.
- Recovery: the retired w4dn8 rollback payload is byte-exact on the Hub —
  rollback is now a **re-download, not a local file**. Corpora (grams,
  kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert
  and KLD-gate inputs.
- Post-checks: frank-archon active and `/health` ok, serving payload
  byte-size intact, the live kvcache tier untouched, no process references
  a deleted path.
- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook
  index rows; the live runbook's Rollback rewritten (HF re-fetch procedure —
  the fetch script's REPO/RAD_NAME/STAGE/DEST/SIZE/SHA256 constants are
  hardcoded, so retargeting at the mirror is a script edit); the retired
  runbook's header corrected (was still "LIVE", never updated at
  retirement).

## 2026-10-17 — frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool

- 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen
  payload with the card's memory-saving KV pool; census first: the n-gram
  table was ALREADY disk-backed and the only real lever was the KV pool.
  See [[frank-dragoon]].
- **Filename trap caught before the fetch:** this repo name carried two
  different artifacts 32 bytes apart (the writer-transplant era's file vs
  the native conversion). Byte-exact gate + LFS sha256 in
  `scripts/download-frank-dragoon-v2abl.sh`, which also HARDLINKS the
  peonist n-gram/vision/tokenizer trio — the release publishes only the
  main file, and a hardlink is a regular file, so the no-symlink serve gate
  passes without a 47.7 GiB copy.
- **Companion risk adjudicated by the engine, not by argument.** The card
  does NOT establish the stock peonist companions as matched to a native
  conversion and publishes no source-matched pair, so the A/B ran the
  checkpoint tools on a stopped service under one knob set
  (`scripts/census-frank-dragoon.sh`) plus a behaviour battery
  (`scripts/battery-frank-dragoon.py`): two sound models disagreeing on
  content, not a corrupt forward pass. The boot window ran through a
  transient `FRANK_DRAGOON_ENV_FILE` drop-in — never a scratch port (one
  iGPU, one server) — with the stop+mask law and a trap-guarded restore
  (`scripts/trial-frank-dragoon.sh`). Masking fails on a relinked unit
  file, as documented; stop alone was enough.
- Cost of the swap, stated plainly: two MAXIMUM-length conversations no
  longer share the endpoint, the finetune's perplexity edge is gone, and
  its reasoning came out compressed ~1.9×. Bought: RAM back as page cache
  for the disk-backed n-gram table, faster cold prefill, clean retrieval,
  and the base checkpoint itself instead of the finetune. The seam's pool
  law relaxed from `pool == slots × ctx` to `ctx ≤ pool ≤ slots × ctx`.

## 2026-10-17 — frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix

- 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704
  (49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet
  needed the bigger window). See [[frank-dragoon]].
- Harness moved WITH the ceiling (400-zone law): `models.yml`
  `contextWindow` = ceiling − `maxTokens`; compaction fires at 85 % of the
  window, and the 180k compaction target of the 262,144 era is unreachable
  because the window itself is smaller. Seam re-keyed to prove the
  invariant at the new pair — `ctx ≤ pool ≤ slots × ctx` and
  `harness window ≤ ctx − maxTokens` — so a future raise of one without the
  other fails the gate.
- Probes on the live endpoint: over the ceiling → `400 … tokens over` in
  0.86 s — preflight rejects before any prefill, so probing costs the
  endpoint nothing.
- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the
  `[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly
  said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex
  Web, signed out in the `qa-e2e` profile, running its own sign-in flow,
  which broke the spec two different ways: (1) it redirects the top frame,
  so Chromium cancelled the crawl's in-loop `page.goto("/")` with
  `net::ERR_ABORTED` — and that reset sat outside the `try`, so one host
  aborted the whole crawl; fixed with `backHome()`, which re-issues on that
  one error class. (2) The same hand-off leaves the browser on
  `app.plex.tv` (Plex's hosted surface for a session-less client), so the
  `hostname == target` law could never settle; fixed with an explicit
  `HANDOFFS` landing list. Plex itself was healthy throughout, and the
  guard contract is still asserted for every vhost. Both symptoms recorded
  as gotcha 6; the gate after both fixes went 33 passed with the crawl
  green on the first attempt.

## 2026-10-18 — issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`

- Ask: does a recent kernel change remove the IOMMU translation penalty
  that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the
  XDNA2 NPU come back? Answered and documented; **no host change** at the
  time — the fix is not installable from the Ubuntu 26.04 archive.
- The change is **AMD PerfOpt** (spec §3.4.9, MMIO `0x016C` bit 13): merged
  via the iommu tree 2026-09-24, first release **Linux 7.4**. It stops
  GPA→SPA translation for the *privileged iGPU* only and leaves the IOMMU
  fully armed for every other device, NPU included. Its only consumer
  requires `AMD_IS_APU` **and the identity domain** — so PerfOpt rides on
  top of `iommu=pt`, it does not replace it.
- **Three premises of the issue corrected.** (1) The 13–16 % is the
  *vendor's* A/B against *their* previous boot line — our record copied the
  figure on 2026-09-11 without ever measuring here. (2) The NPU does **not**
  have to wait for 7.4: `iommu=pt` enables the IOMMU, which is all
  `err=-19` asks for (`amdxdna` binds via `iommu_sva_bind_device()` into its
  own SVA domain, independent of the default domain type). (3) The 460
  tok/s baseline is stale, so the window must measure `off` vs `pt` locally
  and inherit no number.
- Silicon check: our IOMMU is `1022:1508` Strix/Strix Halo rev 02, and the
  **Strix Halo desktop** arm measured only **2–4 %** from PerfOpt (18–23 %
  on Strix Point / Krackan laptops) — the modest arm is the one that
  describes this box. Kernel pick: **skip 6.18–6.18.7**, where an IOMMU
  regression disables SVA outright.
- Self-correction inside the same session: the first pass claimed
  `iommu=pt` alone already avoids the cost. It does not — an identity
  attach keeps the page walk and enables ATS/PRI/PASID/GCR3, so `pt`
  recovers **most** of the gap and PerfOpt is the last part of it. The
  posted comment (Gitea issue #3) was edited to the corrected text.
- Artifacts: a new section in `dankclaw-iommu-translation-off.md`
  (front-matter re-keyed to "vendor-claimed, never measured locally"),
  `frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a
  middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and
  a 4-step maintenance-window procedure that measures locally instead of
  inheriting the vendor number. Nothing deployed — no service was touched.

## 2026-10-18 — boot line flipped: Dankclaw now boots `iommu=pt`

- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the
  research demanded. Provenance first: `/etc/default/grub` had never been in
  the repo while the IOMMU line was the most consequential kernel decision
  on the box, so it is now tracked at `dankclaw/host/grub` (fetched
  byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` —
  diff, install 0644 root:root, `update-grub`, **verify the default
  `grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing
  under `dankclaw/host/`: kernel state is not a container restart.
- **Verdict: identity DMA costs nothing on this host.** Byte-identical
  traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified
  `amd_iommu=off` for five weeks never existed here. Decode's apparent
  −9.7 % was instrument noise: three repeats of the same request on the
  same boot spread 31.8–47.8 t/s.
- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`,
  `/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in
  `render`. No endpoint deployed — revival needs a quadlet with
  `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`.
- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU
  timing for every workload and nothing co-runs on the NPU yet) and any
  PerfOpt work (kernel `7.0.0-38` has zero `perfopt` symbols; the 26.04
  archive caps at `7.0.0-38.38`).
- Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA =
  the Genesys Logic SD reader, a kernel quirk), interrupt remapping on;
  frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy
  edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh`
  rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv`
  is in `~/.local/bin`. Docs re-keyed: the decision record
  `dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md`
  (precondition satisfied, endpoint still absent), AGENTS.md key lesson.

## 2026-10-18 — frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144)
- 2026-10-18 — frank-zealot: 27B PQ2_0 → **Ternary-Bonsai-2-9B PQ2_0 with
  the MTP head**, runtime moved to the PrismML vendor fork build **10777**
  because the MTP arm needs the build that pairs MTP with a quantized V
  cache. See [[frank-zealot]].
- The win is the **window, not the quant**: every conversation goes back at
  the trained 262,144 (the 27B era sold each slot 200,704), and the fleet's
  widest context makes :7771 the **compaction hub** — `models.yml`
  `contextWindow` 200704 → 262144.
- **MTP armed, then OFF by measurement.** The head is in the file and the
  fork loads it, but the draft context bills its own **f16** KV over the
  WHOLE pool, and at `UBATCH_SIZE=2048` that squeeze **halved decode**. The
  documented MTP shape (`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024`) was
  declined — the decode gain is not worth spending the free card on it. The
  arm is recorded in the env header, not left to rediscovery.
- Two bugs the trial caught, both fixed in the repo:
  `${SPEC_PRESENCE_PENALTY:=0.0}` was a *default-assignment*, so the env
  value always won and the documented "MTP runs without
  `--presence-penalty`" arm was unreachable — now `${VAR=0.0}`, which lets
  `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read
  0 for the **entire retained Prometheus history**: llama.cpp answers
  **501** on `/metrics` unless `--metrics` is passed, so the scrape failed
  beside a healthy `/health` and `LlmEndpointDown` fired for days.
  `METRICS=1` is now the default and the seam pins it.
- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the
  first boot was **not the shipped payload**: a leftover
  `frank-zealot.service.d/ptq1.conf` from the bench window pointed
  `FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit
  booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright
  33/33 — **nothing in the gate reads the payload.** Drop-in moved to
  `~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The
  drop-in was not a rogue — see the PTQ1_0 entry below.)
- Measurement law: the first battery reported 117 and 72 t/s because it
  timed natural generations that stopped at 24 tokens — wall-clock aggregate
  on an early EOS reads 4× low. `ignore_eos` + the engine's own `timings`
  numbers is the only honest way to time this endpoint.
- OpenWebUI `model` rows re-keyed in the live DB. Note for the next edit:
  the values live inside the `params`/`meta` JSON columns — the table has no
  `context_length`/`description` columns — and there is no `sqlite3` binary
  on the host, so `python3` stdlib + `Connection.backup()`.

## 2026-10-18 — frank-zealot PTQ1_0 arm: measured, approved, and never written down
- **Correction to the entry above and to my own report.** The
  `frank-zealot.service.d/ptq1.conf` drop-in was NOT a half-finished
  experiment. The session archive (`01a14c93`) carries Frank's own
  instruction at **02:36**: replace frank-zealot with
  `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, use the **Q8** mmproj, batch/ubatch
  **3072/1536**, "fit as much context as possible in 2 contexts".
- It was the production payload **03:14:39 → 03:38:57** and served real
  traffic; recovered from its own `print_timing` lines, PTQ1_0 ran
  ≈47.6–49.0 t/s single-stream greedy decode (25–27 t/s per stream with two
  streams live) and prefilled deep prompts faster than the PQ2_0 27B record
  ever managed.
- Against the PQ2_0 27B record: PTQ1_0 freed the **1,201 MiB** its fetch
  script promised and paid **−37 % decode** — the publisher's "dense
  ternary unpacking loses on Ampere" warning, now confirmed on GA102.
  Frank scratched the whole 27B plan at **03:33** ("scratch the current
  frank-zealot plan completely… 9B… enable mtp… if you can possibly fit a
  third context, do it"), which is what shipped at 13:16.
- The durable failure is mine: those numbers lived only in `/tmp` and the
  journal, never in the repo, so the next agent on the host could not tell
  an approved payload from a rogue drop-in and called it a hijack. The
  fetch script's header now carries the measurement;
  `/home/dank/models/Bonsai2-PTQ1_0/` stays on disk as the 27B rollback and
  the drop-in stays trashed.

## 2026-10-18 — frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot

- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and
  the drop-in check the zealot incident bought ran FIRST: `DropInPaths`
  empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool
  doubled to 401,408 positions over the 200,704 ceiling; the measured boot
  ledger owns the shape. See [[frank-dragoon]].
- Warm reuse survived the raise: one **101,987**-token prompt took 81.5 s
  cold and 0.3 s on repeats — `HALOGEN_CACHE_INPLACE=1` keeps the KV rows in
  the pool, so a warm turn prefills nothing and the doubling evicted nothing
  the cache needed. The RAM price is still real: host `MemAvailable` fell
  **26 → 22 GiB**, because every pool position is RAM the n-gram table's
  page cache loses.
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
  engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured
  figure wins, and every "committed but not yet deployed" clause was retired
  with the measurement (AGENTS.md, the env lore block, both index pages, the
  iommu page, the runbook).
- PTQ1_0 provenance corrected (see the dedicated entry): the arm that
  hijacked the zealot deploy was a **repo-tracked, Frank-ordered, measured**
  candidate — `scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed,
  and its header now carries the recovered numbers (≈48 t/s decode, 801 t/s
  prefill at 141k) and the verdict: −37 % decode against PQ2_0 for 1,201
  MiB freed, then scratched by Frank's 03:33 order in favour of the 9B.
  Nothing is owed on it.
- Census of the halogen HTTP surface while doing it, because the fleet bench
  law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`,
  `/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`,
  `/completion`, `/tokenize`; the chat response carries **no
  `Server-Timing`** header. So the `/completion` `timings` object and the
  `ttft;dur=` header are **llama.cpp** instruments — frank-zealot's, not
  dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine
  owns which instrument.

## 2026-10-18 — frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)
- Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on
  Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile`
  + rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`.
  One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=<name>` is
  honoured only when `/voices/<name>/reference.wav` exists, otherwise a 400
  listing the voices that are — **no silent fallback voice**; oversized
  input is a 414, not a half-narrated sentence; `/health` reports backend +
  voice list; no audio ever reaches the log.
- **Naming law applied (Frank's call):** treat it as a first-class model
  endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`),
  **port 7772**, joining `7770 frank-dragoon` / `7771 frank-zealot`; the
  engine upstream keeps 8880 and is deliberately not a fleet endpoint. The
  internal knob names stay `SHIM_*` (they name the façade component, not the
  service).
- **Voice-pack law recorded** in the env header: a pack needs
  `reference.wav` (24 kHz mono, 18–25 s) **and** the pinned engine's own
  speaker artifact — required, beside the WAV, never replacing it — plus
  `manifest.json` provenance whose offsets are always past the first 10 s of
  the source (the source-material law). Enforcement is split honestly: the
  shim gates on `reference.wav` (the only engine-agnostic term); the pack
  builder is what will refuse to emit a pack without its artifact.
- Landed as **PR #4** (branch `wt/20261017-181502` → `main`, merged as
  `49c4e19`) — the first PR on this repo through the API, which cost three
  422s to get right: the merge endpoint takes **`{"Do":"merge"}`**, `Do`
  being a *string* naming the strategy — `DoMerge`/`merge_method` do not
  exist on this Gitea, and swagger JSON is disabled on the instance, so the
  422 messages are the schema. Law saved to the `gitea-api-pr-operations`
  skill.

## 2026-10-18 — TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion
- Frank listened to the four audition pairs (`qwentts-*.mp3` against their
  `armB-*.mp3` mainline partners) and **passed every clone**. That deleted
  the only axis the A/B could not measure, and the question was reframed:
  **which engine is better for live output**. On that criterion RTF only has
  to stay under 1.0 with margin — the discriminators are time-to-first-audio
  and run-to-run spread, because an arm whose RTF can cross 1.0 stalls a
  live stream mid-sentence.
- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`,
  1.7 B talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on
  8880.** The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env`
  stays in the repo as the rollback. Three consequences are configuration,
  not preference: boot-warm every voice, do not pin a cpuset (threads are
  `hardware_concurrency()/2`), register every pack with its WAV (a lone
  embedding is rejected).
- Two follow-ups deliberately **not** decided: the **0.6 B talker** (never
  auditioned — every clip is 1.7 B) and **ICL vs embedding-only**
  conditioning. Embedding-only is the shipped default because it is cheaper
  and needs no `ref_text`, which this fleet cannot transcribe honestly;
  `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice
  opt-in for batch work where total time beats first byte.
- **Addressing law corrected** in [[Local TTS and Voice Cloning]]: the pair
  may not meet at a host-loopback publish. netavark DNATs published ports
  only and a bridge container cannot reach a host `127.0.0.1` bind — the
  lesson that 502'd hermes — so `host.containers.internal:8880`, which
  `scripts/frank-tts.env` documents, is correct only for a routable bind.
  The deployed pair shares one quadlet network and the façade addresses the
  engine by container name.
- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as
  `1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private
  bridge `10.89.4.0/24`, the API published on **5056** because seerr holds
  `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT` moved 600 → **660** to sit
  one round-trip outside frank-tts's own `SHIM_TIMEOUT`. Merged is not
  runnable: nothing listens on 7772 until the voice stack lands, and the
  episode destination stays interim pending Gitea #7.

## 2026-10-18 — open-notebook: the podcast generator's config lands (nothing deployed)

- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw —
  notes and sources in, a multi-speaker episode out, script written by
  **frank-zealot**, voices rendered by **frank-tts**, no cloud provider
  anywhere in the path. Repo-side only: no host touched. Runbook:
  [[Open Notebook]].
- **Two containers, not one.** `lfnovo/open_notebook:v1-latest` plus
  `surrealdb/surrealdb:v2`, both **digest-pinned** with `Pull=never` —
  frank-dragoon's pinning style, because `:latest` moving underneath a
  rootless service is how you get an unbootable pod after an unrelated
  `systemctl-daemon-reload`. Both health probes are argument-only
  (`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to
  pass without `SURREAL_PASS` in the environment — a credential that must
  never ride the command line, so `Exec=` carries flags only.
- **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` —
  so the API publishes `0.0.0.0:5056:5055`, and because open-notebook's
  frontend derives its API URL from the window origin when `API_URL` is
  unset, `API_URL=http://192.168.0.143:5056` is set to match: a silent
  remap would hang the UI on a port nobody listens on. 8502 publishes on
  `0.0.0.0` because this service carries its own password (the
  episode-audio route is unauthenticated); **8000 is published nowhere**,
  where upstream exposes it on loopback with root credentials, and the seam
  forbids it.
- **The CPU guard, pre-armed.** `frank-tts` spawns one ~8 GiB, 8-thread
  engine per request on a box that also serves two LLMs, while
  open-notebook defaults to a 25-way self-DoS; both knobs are pinned to
  `1` in `scripts/open-notebook.env`. The TTS clock goes to **660 — one
  round-trip outside** frank-tts's own `SHIM_TIMEOUT=600`, so a shim
  give-up reads as the shim's error rather than the worker's; the seam
  asserts the direction (`>`) against the frank-tts env instead of
  restating a number.
- **What a credential needs in order to exist:**
  `OPEN_NOTEBOOK_ENCRYPTION_KEY` must be set before the UI can save *any*
  provider credential, and `OPEN_NOTEBOOK_PASSWORD` must stay **absent** to
  remain auth-off — the app reads `os.environ.get(…) is not None`, so
  `PASSWORD=` boots a login form asking for the empty string. The deployer
  seeds `/home/dank/Containers/open-notebook/env` (600) from the host vault
  and **refuses to start the pair while either secret is blank**, because a
  half-configured open-notebook stores no credentials and fails quietly.
- **Episode destination is interim, by decision.** The destination is
  **Gitea issue #7, `human-only`** — agents must not work it. Until Frank
  settles it, the app keeps its own named volume and
  `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts`
  (`DATA_FOLDER` is hard-coded `./data`), so finished MP3s are ordinary
  files in his home directory instead of volume contents needing
  `podman unshare`. One mount line; one deletion undoes it.
- **Two UI buttons will fail, and that is cosmetic:** Test Connection does
  `GET <base>/models` and the TTS probe synthesises voice `alloy`;
  frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are
  created *by name* and the voice field is free text, so `narrator1/2/3` is
  typed in and never discovered. Documented, not worked around.
- Three deployer seams cost time and are now law-shaped: a service must be
  registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the
  `deploy-remote.sh` dispatch case, its pre-deploy requirement, and
  `deploy.sh`'s dispatch regex, where an omission silently plans an empty
  set for `--only`; a `dankclaw/<svc>/*` case must precede the
  `dankclaw/*.container` glob because that `*` crosses `/`; and the
  changed-path routing lives *inside* the per-path loop, after the
  `scripts/*.env)` case, not at the top of the file.

## 2026-10-18 — frank-tts-engine: the arm A engine becomes deployable (nothing deployed)
- Goal: turn the engine the day's A/B picked into a **repo-owned,
  boot-persistent, self-healing container** — `frank-tts-engine`,
  `qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident
  `tts-server`, container port 8880, and **not** a fleet endpoint: the
  endpoint stays `frank-tts` on 7772. Runbook: [[Frank TTS Engine]].
- **No `PublishPort` at all, which the merged A/B states as law.** netavark
  DNATs published ports only, so a `127.0.0.1:8880` publish is not a path to
  a peer container (the trap that 502'd hermes) — while being the one
  LAN-reachable surface of an unauthenticated synthesiser. Both voice
  containers therefore sit on one committed quadlet network,
  `frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after
  open-notebook's 10.89.4.0/24), the façade addresses the engine by
  container name, and diagnostics — including the deployer's health gate
  (`wait_container_healthy`, there is no URL to dial) — run through
  `podman exec`.
- **The boot warmer is a unit hook, not an operator habit.** The registry is
  memory-only and the first request for a registered voice decodes the whole
  reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine
  /app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what
  stops a failed warm from crash-looping the endpoint under
  `Restart=always`, and it logs `cold=` and `warm=` per voice on every boot.
- **Thread law recorded before anyone tunes it away:** no `--threads`
  exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin
  silently changes the thread count. Nothing pins this container.
- Image builds from the pinned commit with the **proven** line, verbatim —
  `GGML_BLAS=OFF` forced, every GPU backend off, no `-DGGML_NATIVE` because
  ggml's default is `-march=native` for the building host: the image is
  built where it runs. Weights fetched by
  `scripts/download-frank-tts-engine-qwentts.sh` with byte and LFS-oid
  checks; the deployer gates on `--verify-only` rather than pulling 1.4 GiB
  inside a deploy.

## 2026-10-18 — frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down

- Stage 2 of the voice stack: the **façade** in front of the engine is now a
  deployable service too. Runbook: [[Frank TTS]].
- **Two attached networks, in a meaningful order.** `dankclaw_default`
  first — it owns the default route and is where consumers live
  (open-notebook dials `http://frank-tts:7772/v1` by container name) — then
  the voice pair's private bridge. `scripts/frank-tts.env`'s `TTS_UPSTREAM`
  moves from `host.containers.internal:8880` to
  `http://frank-tts-engine:8880`, the actual fix for a shape that could
  never have worked: a bridge container cannot reach a host-loopback bind,
  and the engine publishes nothing.
- **`Wants=` + `After=` on the engine, never `Requires=`:** the façade still
  carries the `llama-tts-cli` rollback backend, which needs no upstream, so
  an engine that will not boot must delay this endpoint, not take it down.
- The façade handler installs the **engine's** quadlet pair as well — not
  tidiness: with a referenced `Network=` file absent the v1 Quadlet
  generator produces no container unit at all — and it `mkdir -p
  /home/dank/voices` as the deploy user, because rootless podman
  auto-creates a missing bind source **root-owned** and the endpoint then
  reports "no voices" instead of an error.
- **A voice is a lowercase directory name.** `^[a-z0-9][a-z0-9._-]*$`,
  ≤ 64 chars, no slash/`..`/space. The shape is the intersection of two
  rules that do not agree: `app.py` would accept mixed case, `tts-server`
  lowercases names — so a pack named `Narrator1` resolves through the façade
  and not through the engine, and the boot warmer refuses such a directory
  loudly. `narrator1/2/3` are load-bearing for open-notebook.
- **No Prometheus job, on purpose.** The façade exports no `/metrics`,
  Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services`
  probes are documented down for exactly that), and `LlmEndpointDown` is
  scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal
  every few minutes for a healthy endpoint. That is the frank-archon
  never-green lesson pointed the other way, written in the page instead of
  quietly ignored.

## 2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code

- Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a
  source recording plus a crop offset into a deployable pack —
  `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own
  `reference.spk` + `reference.rvq` made from it, and `manifest.json`
  provenance. Tracked tunables in `scripts/voice-packs.env`; new runbook
  [[Voice Packs]].
- **The gate is the product.** The project-wide invariant — no voice
  material from the first 10 s of a sample file — had only ever been a
  sentence in `scripts/frank-tts.env`. An offset inside it now exits 1
  before a directory is created, and the same assertion runs again against
  the manifest that was just written, next to the recorded sha256, so a pack
  cannot carry a forbidden crop even if the env file is edited into
  nonsense. `10` stays a constant in the builder; the env file carries
  offsets, never the law.
- **The extractor's contract was read, not guessed:** `qwen-codec` in the
  pinned qwentts.cpp commit has **no `-o`** and names both artifacts next to
  the input with the extension swapped. So the WAV is staged under its final
  stem and `reference.wav` — the file the façade gates on — is moved into
  place **last**, after the artifacts and the manifest: a failed extractor
  leaves a directory that is not a voice. `--force` retires an existing pack
  whole to `.AGENTTRASH/` instead of touching bytes; nothing in this script
  deletes.
- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B
  per 12.5 Hz frame and `enc_dim 2048 × f32` = 8,192 B. That makes the
  **effective crop window 19.84–25 s**, narrower than the WAV law's 18–25 s,
  and a too-short crop is refused before the extractor runs. An engine bump
  that changes either number fails the build by design.

## 2026-10-18 — voice stack audit: three faults that no gate could see
- The voice stack had never been run end to end, so the whole path was
  audited as static code against the tools it calls, and three faults came
  out of it. All three are the same species: a step that **reports success
  while doing nothing**.
- **The build would not have happened.** `Containerfile.engine` created its
  unprivileged user with `adduser`; the pinned ubuntu:24.04 base has no
  `adduser` at all — proven in the base on Dankclaw, not argued from memory.
  Replacement: `useradd --system --uid 10000 --home-dir /app --shell
  /usr/sbin/nologin tts`; its `SYS_UID_MAX` warning is cosmetic and is the
  price of matching the façade's uid.
- **The warm pass would not have warmed.** The unit hands its env file to
  podman via `--env-file`, which performs no shell parsing and takes the
  literal remainder of the line, while `source` strips one quoting layer.
  `WARM_TEXT` therefore reached the container with its quotes, the JSON was
  invalid, and every voice would log "registered, but the warm synthesis
  failed" beside an `active` unit. Fixed in the hook; the bug survived the
  first draft precisely because every existing test used `source`.
- **A rebuilt image would not have been rebuilt.** `build_compose_image`
  skips `podman build` on a matching `cfhash` label, and that hash covered
  the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh`
  (both `COPY`'d) skipped the build while the deploy reported success and
  `/health` answered with old code. Each case arm now hashes the files its
  `Containerfile` copies.
- Docs carry the lessons where an operator will hit them: a **Locally built
  images** section in [[deployment]], two new troubleshooting lines in
  [[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502
  beside 200 = model loaded ≠ voices registered), and `Voice Packs.md`
  step 4 corrected — the warm hook re-registers **every** pack on every run,
  because the engine replaces a name it already holds, so a restart is what
  an engine bump needs and not what a rebuilt pack needs.

## 2026-10-18 — frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap
- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`;
  the image built clean — the first real proof of the `useradd` fix — and the
  unit was `healthy` after 31 s.
- Then the deploy reported `0 voice(s) warm`:
  `/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long`. The
  hook handed the registration body to curl as ONE argument; the kernel caps
  a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB
  `reference.wav` base64's past that, so curl never executed and the hook
  logged that as "HTTP no response". Fourth silent fault of the voice stack,
  same species as the other three: `/health` stayed green throughout. Fix:
  the body goes to a `mktemp` file and curl gets `@file`; the regression test
  is the honest shape — real listener, real `curl`, a 300 KB fixture pack —
  so it fails with the same kernel error the host produced. Every shipped
  pack sits well over the ~96 KB cliff, so no small fixture could ever have
  seen this.
- **A trial law did not survive deployment.** Against an empty registry the
  first synthesis of a freshly registered voice cost ~1.9 s, the next ~1.2 s.
  The trial's 6.2–8.0 s cold / 0.5 ms restore ladder, which the warmer's own
  header repeated, is retired: warming buys registration — mandatory, because
  the registry is memory-only — plus about **0.6 s**. Corrected in
  [[Frank TTS Engine]] law 1 and its troubleshooting table, and in
  [[Local TTS and Voice Cloning]] and [[Voice Packs]].
- Sweep afterwards: the retired ladder still lived in six places the first
  pass missed — both quadlet files, `scripts/frank-tts-engine.env`,
  `scripts/frank-tts.env`, the [[Frank TTS]] troubleshooting row, the
  research brief's consequences paragraph, and one test comment. All
  re-keyed to the measured claim. The trial entry earlier in this log is
  left as written — a dated worklog entry is a record — and this line is the
  correction that points at it.

## 2026-10-18 — 7772 is live and the harness speaks through it
- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16**
  (`d2ce669`, the comment sweep), then deployed both voice services in order
  (`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate
  33/33 twice. Deployed-state proof from a LAN client: `/health` →
  `omni-http` with all three voices, and the same sentence in `narrator1` vs
  `narrator2` came back with different sha256 and different RMS — the name
  actually selects a voice.
- **The harness `speech` role moved off Kokoro**: `config.yml`
  `modelRoles.speech: frank-tts (dankclaw)/frank-tts`. No voice key moves
  with it.
- Two harness laws learned by breaking it first (both now on [[Frank TTS]]):
  an explicit speech role has **no fallback candidate**, so a dead endpoint
  errors instead of degrading; and `models.yml` loads at process start while
  `config.yml` is read live, so a session older than the `frank-tts` row
  fails exactly like a dead endpoint.
- Corrected the same evening, by breaking it in a restarted session: a
  harness call that names no voice reaches 7772 with **no `voice` key at
  all** — `400 unknown voice None; available voices: …` — because no setting
  feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a
  voice-less request speaks the declared pack, an unknown name still 400s
  with the list. `speech.voice: narrator1` — briefly recorded as wiring —
  went back to `am_fenrir`: that key is the vocalizer's Kokoro enum, so a
  pack name there breaks speak-aloud and does nothing for synthesis.
- Status sweep: six pages still said the endpoint did not exist — both vault
  descriptions, the two `index.md` bullets, the Open Notebook blocker (the
  voice leg works; open-notebook itself is what is missing), the engine
  page's deploy order (now a deploy record), and the homepage card, which
  lost `wip: down` so the admin panel probes 7772 for real.

## 2026-10-18 — the default voice shipped: a harness voiceover needs no voice
- Merged **#17** (`b590695`) and **#18** (`7889a68`) and deployed
  `--service frank-tts --service homepage` in one dispatch, Playwright gate
  33/33. The call that had failed is the one that proves the fix: same
  device, **no** `voice_id` — `Saved 430124 bytes (…, voice=default,
  codec=wav)` — and the façade journal says what actually spoke,
  `speech voice=narrator1 (default)`. The harness prints `voice=default`
  because it never had a name to print; the voice that answered is the
  operator's `DEFAULT_VOICE` line in `scripts/frank-tts.env`, and the journal
  keeps them distinguishable forever after.
- The page gained a **Default voice** row in its contract table and
  [[#Naming a voice]] now states that a call naming nothing is the harness's
  normal shape, not a client bug — the misconception that produced the 400 in
  the first place.

## 2026-10-18 — OpenWebUI "voice narrator1 not found": the rejector is in the browser
- Frank's toast — `Error: Voice "narrator1" not found. Should be one of:
  af_heart, …, bm_fable` — looked like frank-tts refusing the name. It is
  not: that 29-name enum is the voice pack of **Kokoro.js running in the
  page** — OpenWebUI 0.11.4 ships that implementation in its own front end.
- Instruments that separated the two paths, in order: the admin config DB is
  correct (`audio.tts.engine=openai`, base `http://host.containers.internal:7772/v1`,
  `voice=narrator1`); from inside the container the exact OpenWebUI body
  returns **200 `audio/mpeg`**; and `podman logs open-webui` holds **zero**
  `audio/speech` requests for the failing click. A server-path failure leaves
  a log line and a frank-tts error — a `browser-kokoro` failure leaves
  neither — and no container env or `user.settings` row carries an override,
  so the engine choice is per-browser storage, not fleet state.
- Fix is a UI click, not a deploy: Settings → Audio → TTS Engine →
  **Default** hands synthesis back to frank-tts and the cloned voices;
  choosing `af_heart` keeps in-browser Kokoro, which cannot clone (fixed
  pack). Recorded on [[OpenWebUI]] under *Text-to-speech (frank-tts)*.
- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so
  `agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a
  plain `omp` session with no flags: `/say merged build check` → WAV to the
  player, temp dir cleaned. Sessions open before the merge need a restart to
  pick the command up.

## 2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV
- 2026-10-18 — frank-zealot: shape cut to 2 × 262,144 over unquantized f16 KV
  (the 786,432 pool cannot fit exact KV on the 3090; concurrency 3 → 2, and
  MTP is now doubly out — off by the 13:16 measurement, and exact KV bills a
  draft-context f16 pool over the whole context). See [[frank-zealot]].
- Arithmetic correction to the repo's long-standing figure: "the f16 pair
  would need **26,112 MiB**" doubled the q8_0 number *including* its 1/32
  scale overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is
  **1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The
  conclusion stands (three f16 windows cannot boot this card); the number was
  wrong.
- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured
  non-KV terms must fit the driver-addressable card with **≥ 1 GiB** of slack
  — and the same gate proves it refuses the impossible arm (f16 at 786,432),
  not merely passes the shipped one.