docs: frank-large Kairic Edge runbook + frank-medium clean-KV updates

- New runbook frank-large-kairic-edge-qwen38-27b.md (ACTIVE frank-large, Kairic Edge runtime, jcbtc Qwen3.8-27B-IU4, 262144 ctx, MTP depth 4 + n-gram drafting, compatibility mode) - AGENTS.md: frank-large section + topology row + fleet contract + runbook index (Kairic Edge ACTIVE, unsloth RETIRED same-day); frank-medium section body (MXFP4 + unquantized bf16 KV, 486,464-token pool) - LLM index + top-level index: Kairic Edge ACTIVE, unsloth RETIRED, system specs entry
2026-09-21 19:16:51 -07:00
parent b2a74004df
commit e04f510883
4 changed files with 282 additions and 69 deletions
@@ -0,0 +1,195 @@
+---
+type: reference
+title: Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
+description: The active frank-large runbook — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing.
+tags:
+  - llm
+  - kairic-edge
+  - rocmfpx
+  - dankclaw
+  - frank-large
+  - strix-halo
+  - gfx1151
+  - iu4
+  - w4a4
+  - rocm
+  - mtp
+  - promptforge
+timestamp: 2026-09-21
+---
+
+# Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
+
+> **ACTIVE since 2026-09-21.** It replaced the unsloth llama.cpp prebuilt
+> (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE — see
+> [[frank-large-unsloth-qwen38-flash-next]], the current rollback target),
+> which had replaced the CIRU v4.4.1 Orca runtime (see
+> [[frank-large-ciru-orca]]). Deployed as a **root systemd unit**
+> (`scripts/frank-large.service`, `User=dank`), port 7770, on the same
+> gfx1151 iGPU. The RTX 3090 it does not touch belongs to
+> [[frank-small-gemma4-12b]].
+
+The model is `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` — a **27B dense** Qwen3.8
+at **IU4** (W4A4, integer-weights / integer-activations). It is a
+fundamentally different architecture from the 125B MoE it replaces: no
+PLE, no MoE experts, no n-gram embedding. The payload is four files: the
+15.5 GiB main GGUF plus three `.pfs` sidecars (FFN 7.99 GiB, GDN 1.88
+GiB, GDN-Output 0.70 GiB) — the PromptForge IU4 weight shards the Kairic
+Edge runtime loads alongside the main GGUF.
+
+The runtime is the **Kairic Edge** build of `ciru-ai/ROCmFPX` (branch
+`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`), built from source
+(`scripts/build-kairic-edge-gfx1151.sh`). It is a llama.cpp build with the
+PromptForge sidecar env block and the `--kairic-edge` qualified serving
+path.
+
+## Why this swap
+
+- The operator asked for the Kairic Edge 27B model at **one conversation,
+  full context (262,144)** (2026-09-21).
+- The Kairic Edge runtime is the qualified IU4 (W4A4) serving path — no
+  standard llama.cpp quant, no PLE sidecar payload, no QSA hotfix: one
+  GGUF, three sidecars, done.
+- The spec method is native MTP depth 4 + n-gram drafting (24/64/64).
+  DFlash is NOT available for this model (no `DRAFT_DFLASH` spec type in
+  the Kairic Edge runtime).
+
+## Shape
+
+One root unit, one process, one published port:
+
+| half | port | reachable |
+|---|---|---|
+| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
+
+## Runtime
+
+- **Source build**: `scripts/build-kairic-edge-gfx1151.sh` (TheRock
+  7.15.0a20260718 toolchain, AMD clang 23.0.0, HIP 7.15.0, GCC 13.3.0,
+  CMake 4.4.2, Ninja 1.13.0). Output at
+  `/home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server`.
+- **Composable Kernel**: `fdf4bb7fcc984811cef48ce817d89aac064b984a` +
+  `patches/composable-kernel-gfx1151-iu4.patch` (SHA-256
+  `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5`).
+  The CK `config.h` is generated by CMake's `configure_file` (line 334 of
+  CK's CMakeLists.txt) — it is untracked, so a `git clean` removes it.
+- **GPU target**: `gfx1151` (Strix Halo). The build takes ~4 minutes
+  (`JOBS=$(nproc)`).
+- **v1.2 changes**: Exact 65-row n-gram verification switched to the compact
+  authoritative path by default; the native IU4 M65 verifier is retained
+  only for diagnostics.
+
+## Spec method
+
+- **Native MTP depth 4** (`--spec-type draft-mtp --spec-draft-n-max 4`):
+  the model's MTP layers draft 4 tokens per step.
+- **N-gram drafting** (24/64/64): the `ngram-mod` matcher chains behind
+  the MTP drafter (the `--spec-type draft-mtp` value builds both).
+- **DFlash is NOT available**: the Kairic Edge runtime's spec-type
+  constants are `DRAFT_EAGLE`, `DRAFT_MTP`, `DRAFT_SIMPLE`,
+  `NGRAM_CACHE`, `NGRAM_MAP_K`, `NGRAM_MOD`, `NGRAM_SIMPLE`, `NONE` — no
+  `DRAFT_DFLASH`. The z-lab DFlash2-FP8 drafter (frank-medium's drafter)
+  cannot be used.
+- **Draft device**: `--spec-draft-device ROCm0 --spec-draft-ngl 999`
+  (the drafter runs on the iGPU, all layers).
+
+## Compatibility mode
+
+The Kairic Edge runtime has two modes:
+
+- **Fast-greedy** (`KAIRIC_EDGE_COMPATIBILITY_MODE=0`, the runner's
+  default): rejects request features that require full target logits,
+  including sampling, penalties, probabilities, grammar-constrained tool
+  calls, logit bias, LoRA, and a reasoning budget.
+- **Compatibility** (`KAIRIC_EDGE_COMPATIBILITY_MODE=1`): enables all of
+  the above (9.7% slower per the build guide).
+
+The harness (OMP) needs sampling, tools and a reasoning budget, so
+`COMPATIBILITY_MODE=1` is the deployed shape. In compatibility mode:
+`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0` and
+`PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0`.
+
+**Known gotcha**: the greedy-argmax fastpath (fast mode) rejects
+non-greedy requests and reasoning budgets. Re-test chat before production
+if you flip to fast mode.
+
+## PromptForge sidecars
+
+The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
+(the certified runner's exact env):
+
+| sidecar | env var | size |
+|---|---|---|
+| FFN | `PROMPTFORGE_IU4_SIDECAR` / `PROMPTFORGE_SIDECAR` | 7.99 GiB |
+| GDN | `PROMPTFORGE_GDN_SIDECAR` | 1.88 GiB |
+| GDN-Output | `PROMPTFORGE_GDN_OUTPUT_SIDECAR` | 0.70 GiB |
+
+The serve script (`scripts/serve-frank-large.sh`) validates binary → main
+GGUF → 3 sidecars and exits with a pointer message if any is missing, then
+`exec`s `llama-server` with `--jinja --alias frank-large
+--chat-template-kwargs '{"preserve_thinking":true}'` (Qwen3.8's template
+needs this to carry thinking blocks across turns; a request-level value
+still wins).
+
+## Memory budget
+
+The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
+`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
+"dedicated" is not the budget). The 27B dense model (26 GiB total) is
+mmap'd in with `-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV +
+buffers), leaving ~84 GiB of aperture headroom. The 8 GiB prompt-cache
+budget (`CACHE_RAM=8192`) is the Kairic Edge qualified config. The model
+is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet `CACHE_RAM`
+floor does not apply.
+
+## Payload
+
+`scripts/download-frank-large-kairic.sh` (resumable, each file verified
+against its declared byte count + sha256; `--verify-only` re-checks):
+
+| file | size | sha256 |
+|---|---|---|
+| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 16,617,792,672 | `360caf73…` |
+| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 8,576,856,064 | `adcbb90a…` |
+| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 2,019,569,664 | `82f93131…` |
+| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 756,953,088 | `3b07e7b1…` |
+
+## Client contract
+
+- **Port**: 7770 (the same port and alias the unsloth predecessor served).
+- **Alias**: `frank-large`.
+- **Context**: 262,144 (the model's native window), 1 slot.
+- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
+  --reasoning-budget -1`), extracted to `reasoning_content`.
+- **Vision**: text-only (the model has no vision projector). The fleet's
+  image path is [[frank-small-gemma4-12b]] either way.
+- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
+  {xhigh, medium, low} — the harness folds its levels onto that set in
+  `models.yml` (`reasoningEffortMap`).
+
+## Measured performance
+
+10-question conversation (2026-09-21, post-deploy): **23.99s total, 530
+tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
+MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
+`ngram-mod` and `draft-mtp` active in the journal).
+
+## Failure modes worth knowing
+
+- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes
+  it, and the next build fails. Re-run the build script (it re-checks-out
+  the CK commit and re-applies the patch).
+- **Fast-greedy mode rejects the harness**: if you flip to
+  `COMPATIBILITY_MODE=0`, the harness's sampling/tools/reasoning-budget
+  requests are rejected. Re-test chat before production.
+- **The `charlie12345` fork is NOT the Kairic Edge repo**: the local
+  clone at `/home/dank/GitPackages/ROCmFPX` is the `charlie12345/ROCmFPX`
+  fork (no kairic branch). The Kairic Edge build uses the `ciru` remote
+  (`https://github.com/ciru-ai/ROCmFPX.git`), branch
+  `kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`, in the worktree at
+  `/home/dank/GitPackages/ROCmFPX-kairic/`.
+
+## Rollback
+
+[[frank-large-unsloth-qwen38-flash-next]] — the unsloth prebuilt payload
+and runtime are intact on disk.
@@ -4,10 +4,11 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-21):
 **vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
 process across both Radeon AI PRO R9700 cards (just1moremodel
 Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
-R4D one-shot allreduce), and TWO endpoints on Dankclaw — the unsloth
-llama.cpp prebuilt (frank-large, Qwen3.8-Flash-Next UD-IQ4_XS on the gfx1151
-iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
-Q4_K_M with MTP and vision on the RTX 3090, port 7771).
+R4D one-shot allreduce), and TWO endpoints on Dankclaw — the Kairic Edge
+runtime (frank-large, ciru-ai/ROCmFPX v1.2, jcbtc Qwen3.8-27B-IU4-Kairic-Edge
+IU4 W4A4 on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
+HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port
+7771).

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -31,10 +32,10 @@ ONE process on both cards, 262144 ctx, up to 8 seqs]
    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
 gfx1151 iGPU 114 GiB GTT]
-        E[unsloth llama.cpp b11030 (ROCm gfx1151)
+        E[Kairic Edge (ciru-ai/ROCmFPX v1.2, gfx1151)
 frank-large, port 7770]
-        F[Qwen3.8-Flash-Next UD-IQ4_XS
-200000 ctx / 1 slot, MTP depth 2]
+        F[Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4)
+262144 ctx / 1 slot, MTP depth 4 + n-gram]
        G[mainline llama.cpp 37b53fd
 frank-small, port 7771]
        H[Gemma4-12B QAT Q4_K_M
@@ -95,11 +96,18 @@ frank-small, port 7771]
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-unsloth-qwen38-flash-next]] — ACTIVE frank-large runbook
-  since 2026-09-21 (root systemd unit, port 7770): the unsloth llama.cpp
-  prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
-  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the gfx1151 iGPU — one
-  200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
+- [[frank-large-kairic-edge-qwen38-27b]] — ACTIVE frank-large runbook
+  since 2026-09-21 (root systemd unit, port 7770): the Kairic Edge runtime
+  (ciru-ai/ROCmFPX v1.2, built from source) serving
+  jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense) on the gfx1151
+  iGPU — one full 262,144-token conversation, MTP depth 4 + n-gram drafting,
+  the three PromptForge sidecars, compatibility mode, the 114 GiB GTT memory
+  model, client contract, troubleshooting
+- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
+  swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
+  the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
+  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
+  one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
  head, the 114 GiB GTT memory model, client contract, troubleshooting
 - [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
@@ -167,7 +175,7 @@ frank-small, port 7771]
  (deployed and reverted 2026-08-22)
 - [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
 - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
-  unsloth llama.cpp prebuilt since 2026-09-21, frank-small on the RTX 3090)
+  Kairic Edge runtime since 2026-09-21, frank-small on the RTX 3090)
 - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
  on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
  the retirement record — the IOMMU/SVA err=-19 root cause and where
@@ -147,12 +147,19 @@ The LLM inference fleet on the two hosts. Full index:
  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
  Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
  rollback record)
- [[frank-large-unsloth-qwen38-flash-next]] — the ACTIVE frank-large runbook
-  since 2026-09-21 (root systemd unit, port 7770): the unsloth llama.cpp
-  prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
-  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the gfx1151 iGPU — one
-  200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
-  head, the 114 GiB GTT memory model, client contract
+- [[frank-large-kairic-edge-qwen38-27b]] — the ACTIVE frank-large runbook
+  since 2026-09-21 (root systemd unit, port 7770): the Kairic Edge
+  runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
+  Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B
+  dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151
+  iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram
+  drafting, compatibility mode, the 114 GiB GTT memory model, client contract
+- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (replaced by
+  [[frank-large-kairic-edge-qwen38-27b]]; same-day swap): the unsloth
+  llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
+  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the
+  gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the
+  explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
 - [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 →
  09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving