log: record the 2026-09-15/16 ik cutover and decode campaign

serverAppConfigs-docs/log.md

@@ -498,3 +498,46 @@ timestamp: 2026-09-14
   no WAN path to Sonarr at all — manual DNS could previously reach the vhost,
   which served the zero-auth UI to anyone who resolved the name manually at
   the public edge; the vhost (added in `e0bade6`, 2026-09-04) is retired.
+
+## 2026-09-15/16 — frank-large: ik_llama.cpp cutover + the decode campaign
+- Deployed (0c1e10a, via clean-worktree stash cycle around the peer's
+  uncommitted files): halogen quadlet retired to
+  `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet`, frank-large now a root
+  systemd unit (`scripts/frank-large.service` → `/etc/systemd/system/`)
+  running the committed serve script. Service verified live: health 200,
+  reasoning completions, vision, slot persistence.
+- Overnight performance campaign against the halogen bar (Frank: ~30 t/s
+  decode). Result: decode stuck at 3.8-4.3 t/s across every safe
+  configuration; prefill ~192 t/s = the CPU AVX512-VNNI ceiling. Full
+  evidence chain in [[frank-large-ik-uncensored]]: (1) ~100 CPU↔GPU backend
+  splits per token with the 3090 at 5% SM utilization — the decode wall is
+  GPU-section launch/sync overhead, not compute or memory placement;
+  (2) T1 ≈ T16 decode (3.97 vs 3.8-4.0) — the wall is a serial critical
+  path; (3) ik's CUDA-graph capture aborts on qwen4exp's per-layer
+  `ggml_repeat_4d` hc-broadcast ops; (4) mainline llama.cpp `37b53fd` runs
+  1.0 t/s on this model (CPU MoE kernels 4x slower than ik's iqk) — ruled
+  out; (5) the GGUF has no MTP head (mradermacher stripped it) — no
+  speculative decoding; (6) 84 GiB of experts cannot fit 24 GiB VRAM, so
+  halogen's all-resident 30 t/s is physically out of reach for this model
+  on this host. Re-evaluate when mainline's hc-ops + graph handling mature
+  (ggml-org PR #28901).
+/- Measurement integrity traps hit live: the systemd unit's Restart=always
+  respawned pkill'd servers mid-experiment (phantom instances answering
+  health — manual experiments must `sudo systemctl stop frank-large.service`
+  first; plain `systemctl --user` lines are stale halogen-era docs); a
+  concurrent `-j16` build starved a decode measurement into a false timeout;
+  warm-cache `cat` benchmarks invalidated a concurrent perf window.
+- `-ncmoe 42` (+6 GPU expert layers) measured 4.54 t/s decode at 22.0/24.6
+  GiB VRAM — documented as a tuning lever, not shipped (OOM margin at 262k
+  ctx is operator's call). `-ncmoe 48 -thp` at 4.6 t/s holds a 110 GiB
+  anonymous host buffer — co-tenant killer, never ship.
+- Trash staged for approval: dankclaw `~/.AGENTTRASH/20260916-ik-partfiles/`
+  (111 GiB, the 3 GGUF download parts, superseded by the concatenated
+  verified GGUF), `~/.AGENTTRASH/20260915-ik-swap/` (halogen quadlet),
+  repo `tests/.AGENTTRASH/frank_large_quadlet.bats` (retired seam),
+  `scripts/.AGENTTRASH/frank-small*` (earlier retirement).
+- Docs: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` (new
+  runbook), LLM + root indexes relinked, AGENTS.md topology row +
+  Active Services section + bench-policy unit type. Committed 795b0b3;
+  tests 300 ok with the single pre-existing not-ok-73 (comfyui selector
+  bug, unrelated).