frank-large: document the ik_llama.cpp deployment and the decode performance campaign

- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
  decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
  root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
  deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
  halogen-flash-server retired)

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 |
-| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai/halogen-qwen3.8-flash-next w4b .hgn, 262144 ctx / 3 slots, vision tower on | 7770 |
+| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
 
 Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -192,8 +192,8 @@ test process cannot share the single GPU anyway; validate against the service
 itself.
 
 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (rootless — no sudo):
-     `systemctl --user stop frank-large && systemctl --user mask frank-large`
+   - Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover):
+     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankserver `frank-medium` (system unit):
      `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
 2. Bench.
@@ -224,38 +224,29 @@ input anywhere in the fleet** (left with `frank-npu`).
 
 ## Active Services
 
-### frank-large — halogen-flash-server (Dankclaw)
+### frank-large — ik_llama.cpp (Dankclaw)
 
-`ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a rootless Quadlet
-(`dankclaw/frank-large/quadlet/frank-large.container` → `frank-large.service`
-in the user manager), serving `peonist-ai/halogen-qwen3.8-flash-next`.
-One container, two halves: engine on container loopback :8730 (unauthenticated,
-NEVER published) and OpenAI front-end on :8731 published as host **7770**.
+ik_llama.cpp master `d85a1d7` as a **root systemd unit** (repo
+`scripts/frank-large.service` symlinked into `/etc/systemd/system/`;
+`Restart=always`), serving `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
+(111 GiB GGUF + mmproj-f16) on port 7770 with 262144 ctx / 1 slot, greedy
+default, reasoning on (deepseek), vision via mmproj. Topology: the RTX 3090
+(PCIe x4 in the M.2 dock) carries dense/attention/KV; ALL MoE expert tensors
+and the ~33 GiB PLE table stay in LPDDR via CPU buffers. Deployed flags in
+`scripts/frank-large.env` / `scripts/serve-frank-large.sh`; the serve script
+refuses to start without `GGML_CUDA_NO_PINNED_WEIGHTS=1` (the pinned-host
+buffer incident starved co-tenants into unreachability until a power cycle).
 
-Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`,
-`HALOGEN_KV_SLOTS=3`, `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`,
-`HALOGEN_VISION_TOWER=1`. Everything else upstream default. Sizing history
-and the flag-by-flag rationale live in the runbook — do not re-derive here.
+Measured envelope (2026-09-15/16 campaign): prefill ~192 tok/s (1.8k-32k
+prompts, CPU VNNI compute ceiling), decode ~3.8-4.3 tok/s — gated by ~100
+CPU↔GPU backend splits per token (GPU 95% idle, launch/sync-bound; thread
+count irrelevant). The halogen-era 30 tok/s decode bar is out of reach on
+this model+hardware; the runbook carries the full evidence chain, the
+mainline-llama.cpp comparison (1.0 tok/s — ruled out), and the tuning levers.
 
-Invariants that survive any config change:
-
-- **Pool and reserve are a matched pair.** Budget = `121.2 host − 67.7
-  weights − reserve`; that one budget carries KV pool + prefill arena + the
-  1.97 GiB vision tower. The reserve is **not an allocation** — raising it
-  cannot free host RAM, it can only shrink the pool. To free host RAM, cut
-  `HALOGEN_KV_POOL_POSITIONS` or `HALOGEN_MAX_TOK`.
-- **`free` lies on this host.** The ~68 GiB pinned weights count as
-  reclaimable page cache, so `free`/`MemAvailable`/Grafana overstate free
-  memory by the model size. Believe the server's startup ledger. A co-tenant
-  running the box dry looks like minutes of full-CPU stalls with no disk
-  activity — not a crash.
-- **Client-visible contract:** no server-side sampler defaults (no
-  `temperature` ⇒ greedy); per-request budget defaults to 8192 and **includes
-  thinking** (running out mid-thought ⇒ `finish_reason: "length"`, empty
-  content); cap 65536; a request reserving prompt + `max_tokens` beyond the
-  pool queues, then fails `503 engine_busy` after the 3600 s timeout.
-
-Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
+Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
+Rollback: [[halogen-flash-server]] (quadlet archived in
+`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
 
 ### frank-medium — ROCmFPX llama.cpp (Dankserver)
 

serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

@@ -0,0 +1,120 @@
+---
+type: reference
+description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
+tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance]
+timestamp: 2026-09-16T03:30:00-07:00
+---
+
+# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
+
+**ACTIVE since 2026-09-15.** Replaced [[halogen-flash-server]] (rollback target;
+quadlet unit archived in `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` on
+dankclaw). Serves `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
+(111 GiB GGUF + mmproj-f16) at `192.168.0.143:7770`.
+
+## Shape
+
+- Runtime: ik_llama.cpp master `d85a1d7`, built at
+  `/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
+  -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
+  -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
+- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service`
+  symlinked to `scripts/frank-large.service` in the repo; enabled at boot;
+  `Restart=always`. Serve script `scripts/serve-frank-large.sh` + env
+  `scripts/frank-large.env`.
+- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
+  dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
+  ~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
+- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek
+  format, unlimited budget), vision via mmproj, slot-state persistence under
+  `scripts/slot-state`, 262144 ctx, 1 slot.
+
+## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)
+
+| config | decode tok/s | prefill tok/s | verdict |
+|---|---|---|---|
+| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped |
+| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads |
+| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile |
+| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** |
+| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) |
+| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM |
+| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk |
+
+- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling**
+  for CPU-resident experts (expert weight reads amortize across the batch;
+  compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore
+  unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
+- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU
+  backend splits per token (`graph splits = 100` in the log); the 3090 sits at
+  **5% SM utilization** during decode; thread count does not matter (T1 ≈
+  T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.
+
+## Root cause chain (perf evidence)
+
+1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in
+   `ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause`
+   instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%.
+2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on
+   a serial critical path, not parallel overhead.
+3. GPU 95% idle during decode ⇒ the critical path is the GPU-section
+   launch/sync overhead per split (x4 dock link, eager kernel launches).
+4. ik's CUDA-graph capture would collapse those launches but **aborts** on
+   qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`,
+   4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose
+   CUDA path does a capture-illegal operation
+   (`CUDA error: operation not permitted when stream is capturing` → abort).
+5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all
+   land at 3.8-4.6 tok/s.
+
+## Why the 30 tok/s halogen bar is out of reach (honest record)
+
+Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR
+(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must
+split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public
+runtime pays the split penalty above. Paths to >30 tok/s, in order of
+realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops —
+mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops");
+re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on
+GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.
+
+## Upstream issues tracking the slow TG path
+
+ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433
+(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative
+decoding is unavailable for this GGUF — mradermacher stripped the NextN head
+(verified: no `blk.48.*`/nextn tensors in the file).
+
+## Traps (learned the hard way 2026-09-15/16)
+
+- **Pinned host buffers strand the host**: a default-policy start
+  page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and
+  starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the
+  serve-script guard exist for this. `-ncmoe N` with default NO_PINNED
+  settings produces the same 108 GiB pinned buffer — never run it without the
+  guard.
+- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load
+  (`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed
+  CPU buffer" comment in the serve script is wrong on the mmap word; the
+  buffer is anonymous. All placement variants perform alike anyway.
+- **`--defer-experts` does not keep experts off the GPU** under
+  `-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with
+  `cudaMalloc failed: out of memory` at ~79.7 GiB on device 0.
+- **The systemd unit self-heals through manual tests**: `pkill` the server and
+  the unit respawns it ~4 s later (Restart=always) — two instances, port
+  races, phantom measurements. For manual experiments:
+  `sudo systemctl stop frank-large.service` first (it is a root unit; plain
+  `systemctl --user` lines in older docs are stale), and restore with
+  `sudo systemctl start frank-large.service`.
+- Verify "HTTP server listening" in the CURRENT log and
+  `pgrep -xc llama-server` == 1 before trusting any measurement.
+- FA must stay ON — FA-off crashes the loader on this arch.
+- Do not run builds concurrently with decode measurements (`-j16` starves the
+  service and corrupts both results).
+
+## Rollback
+
+Stop and disable the root unit, trash it, restore the halogen quadlet from
+`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
+`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
+the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.

serverAppConfigs-docs/LLM/index.md

@@ -1,10 +1,10 @@
 # LLM Fleet — Knowledge Base
 
 Documentation for the LLM inference fleet on the two hosts
-(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
+(2026-09-16): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
 kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and
-halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai
-Qwen3.8-Flash-Next w4b).
+ik_llama.cpp on Dankclaw (frank-large, mradermacher Qwen3.8-Flash-Next-
+Uncensored i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid).
 
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -26,11 +26,12 @@ layer split, 131k ctx / 1 slot, MTP]
     end
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
-        E2[gfx1151 iGPU 64 GiB unified memory]
-        E[halogen-flash-server 0.5.8
+        E2[RTX 3090 (PCIe x4 dock) +
+gfx1151 iGPU 64 GiB unified memory]
+        E[ik_llama.cpp master d85a1d7
 frank-large, port 7770]
-        F[peonist-ai Qwen3.8-Flash-Next
-w4b .hgn + overlay, 262k ctx / 4 slots]
+        F[mradermacher Qwen3.8-Flash-Next-Uncensored
+i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision]
         E2 --> E --> F
     end
 
@@ -58,8 +59,13 @@ w4b .hgn + overlay, 262k ctx / 4 slots]
   (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11:
-  peonist-ai/halogen-flash-server 0.5.8 serving
+- [[frank-large-ik-uncensored]] — ACTIVE frank-large runbook since
+  2026-09-15: ik_llama.cpp (root systemd unit) serving mradermacher/
+  Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/
+  KV) + LPDDR (experts/PLE) hybrid; measured performance envelope, the
+  decode root-cause chain, tuning levers, and the co-tenancy traps
+- [[halogen-flash-server]] — RETIRED 2026-09-15 (rollback target): the
+  peonist-ai halogen-flash-server 0.5.8 runbook serving
   peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn +
   quality overlay), rootless Quadlet container on Dankclaw port 7770 —
   deployed flags, memory model, client contract, troubleshooting

serverAppConfigs-docs/index.md

@@ -125,8 +125,12 @@ The LLM inference fleet on the two hosts. Full index:
   agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
   Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
   rollback record)
-- [[halogen-flash-server]] — the ACTIVE frank-large runbook since
-  2026-09-11: peonist-ai/halogen-flash-server 0.5.8 serving
+- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since
+  2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-
+  Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd
+  unit, port 7770) — measured performance envelope and traps
+- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and
+  rollback target: peonist-ai/halogen-flash-server 0.5.8 serving
   peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
   container on Dankclaw, port 7770 — flags, memory model, client contract
 - [[dankclaw-iommu-translation-off]] — why Dankclaw boots with