Frank
97f60b35ef
The serve script now passes --chat-template-file pointing at the Qwen-Sharp v22.4.0 template. Without it the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request. With the sharp template wired in, the same request produces a 62-175-token prompt.
213 lines
9.4 KiB
Markdown
213 lines
9.4 KiB
Markdown
---
|
||
type: reference
|
||
title: Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
|
||
description: The active frank-large runbook — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing.
|
||
tags:
|
||
- llm
|
||
- kairic-edge
|
||
- rocmfpx
|
||
- dankclaw
|
||
- frank-large
|
||
- strix-halo
|
||
- gfx1151
|
||
- iu4
|
||
- w4a4
|
||
- rocm
|
||
- mtp
|
||
- promptforge
|
||
timestamp: 2026-09-21
|
||
---
|
||
|
||
# Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
|
||
|
||
> **ACTIVE since 2026-09-21.** It replaced the unsloth llama.cpp prebuilt
|
||
> (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE — see
|
||
> [[frank-large-unsloth-qwen38-flash-next]], the current rollback target),
|
||
> which had replaced the CIRU v4.4.1 Orca runtime (see
|
||
> [[frank-large-ciru-orca]]). Deployed as a **root systemd unit**
|
||
> (`scripts/frank-large.service`, `User=dank`), port 7770, on the same
|
||
> gfx1151 iGPU. The RTX 3090 it does not touch belongs to
|
||
> [[frank-small-gemma4-12b]].
|
||
|
||
The model is `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` — a **27B dense** Qwen3.8
|
||
at **IU4** (W4A4, integer-weights / integer-activations). It is a
|
||
fundamentally different architecture from the 125B MoE it replaces: no
|
||
PLE, no MoE experts, no n-gram embedding. The payload is four files: the
|
||
15.5 GiB main GGUF plus three `.pfs` sidecars (FFN 7.99 GiB, GDN 1.88
|
||
GiB, GDN-Output 0.70 GiB) — the PromptForge IU4 weight shards the Kairic
|
||
Edge runtime loads alongside the main GGUF.
|
||
|
||
The runtime is the **Kairic Edge** build of `ciru-ai/ROCmFPX` (branch
|
||
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`), built from source
|
||
(`scripts/build-kairic-edge-gfx1151.sh`). It is a llama.cpp build with the
|
||
PromptForge sidecar env block and the `--kairic-edge` qualified serving
|
||
path.
|
||
|
||
## Why this swap
|
||
|
||
- The operator asked for the Kairic Edge 27B model at **one conversation,
|
||
full context (262,144)** (2026-09-21).
|
||
- The Kairic Edge runtime is the qualified IU4 (W4A4) serving path — no
|
||
standard llama.cpp quant, no PLE sidecar payload, no QSA hotfix: one
|
||
GGUF, three sidecars, done.
|
||
- The spec method is native MTP depth 4 + n-gram drafting (24/64/64).
|
||
DFlash is NOT available for this model (no `DRAFT_DFLASH` spec type in
|
||
the Kairic Edge runtime).
|
||
|
||
## Shape
|
||
|
||
One root unit, one process, one published port:
|
||
|
||
| half | port | reachable |
|
||
|---|---|---|
|
||
| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
|
||
|
||
## Runtime
|
||
|
||
- **Source build**: `scripts/build-kairic-edge-gfx1151.sh` (TheRock
|
||
7.15.0a20260718 toolchain, AMD clang 23.0.0, HIP 7.15.0, GCC 13.3.0,
|
||
CMake 4.4.2, Ninja 1.13.0). Output at
|
||
`/home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server`.
|
||
- **Composable Kernel**: `fdf4bb7fcc984811cef48ce817d89aac064b984a` +
|
||
`patches/composable-kernel-gfx1151-iu4.patch` (SHA-256
|
||
`ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5`).
|
||
The CK `config.h` is generated by CMake's `configure_file` (line 334 of
|
||
CK's CMakeLists.txt) — it is untracked, so a `git clean` removes it.
|
||
- **GPU target**: `gfx1151` (Strix Halo). The build takes ~4 minutes
|
||
(`JOBS=$(nproc)`).
|
||
- **v1.2 changes**: Exact 65-row n-gram verification switched to the compact
|
||
authoritative path by default; the native IU4 M65 verifier is retained
|
||
only for diagnostics.
|
||
|
||
## Spec method
|
||
|
||
- **Native MTP depth 4** (`--spec-type draft-mtp --spec-draft-n-max 4`):
|
||
the model's MTP layers draft 4 tokens per step.
|
||
- **N-gram drafting** (24/64/64): the `ngram-mod` matcher chains behind
|
||
the MTP drafter (the `--spec-type draft-mtp` value builds both).
|
||
- **DFlash is NOT available**: the Kairic Edge runtime's spec-type
|
||
constants are `DRAFT_EAGLE`, `DRAFT_MTP`, `DRAFT_SIMPLE`,
|
||
`NGRAM_CACHE`, `NGRAM_MAP_K`, `NGRAM_MOD`, `NGRAM_SIMPLE`, `NONE` — no
|
||
`DRAFT_DFLASH`. The z-lab DFlash2-FP8 drafter (frank-medium's drafter)
|
||
cannot be used.
|
||
- **Draft device**: `--spec-draft-device ROCm0 --spec-draft-ngl 999`
|
||
(the drafter runs on the iGPU, all layers).
|
||
|
||
## Compatibility mode
|
||
|
||
The Kairic Edge runtime has two modes:
|
||
|
||
- **Fast-greedy** (`KAIRIC_EDGE_COMPATIBILITY_MODE=0`, the runner's
|
||
default): rejects request features that require full target logits,
|
||
including sampling, penalties, probabilities, grammar-constrained tool
|
||
calls, logit bias, LoRA, and a reasoning budget.
|
||
- **Compatibility** (`KAIRIC_EDGE_COMPATIBILITY_MODE=1`): enables all of
|
||
the above (9.7% slower per the build guide).
|
||
|
||
The harness (OMP) needs sampling, tools and a reasoning budget, so
|
||
`COMPATIBILITY_MODE=1` is the deployed shape. In compatibility mode:
|
||
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0` and
|
||
`PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0`.
|
||
|
||
**Known gotcha**: the greedy-argmax fastpath (fast mode) rejects
|
||
non-greedy requests and reasoning budgets. Re-test chat before production
|
||
if you flip to fast mode.
|
||
|
||
## PromptForge sidecars
|
||
|
||
The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
|
||
(the certified runner's exact env):
|
||
|
||
| sidecar | env var | size |
|
||
|---|---|---|
|
||
| FFN | `PROMPTFORGE_IU4_SIDECAR` / `PROMPTFORGE_SIDECAR` | 7.99 GiB |
|
||
| GDN | `PROMPTFORGE_GDN_SIDECAR` | 1.88 GiB |
|
||
| GDN-Output | `PROMPTFORGE_GDN_OUTPUT_SIDECAR` | 0.70 GiB |
|
||
|
||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||
GGUF → 3 sidecars and exits with a pointer message if any is missing, then
|
||
`exec`s `llama-server` with `--jinja --alias frank-large
|
||
--chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file
|
||
$TEMPLATE_FILE` (Qwen3.8's template needs `preserve_thinking` to carry
|
||
thinking blocks across turns; a request-level value still wins).
|
||
|
||
## Chat template
|
||
|
||
The serve script passes `--chat-template-file` pointing at
|
||
`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
|
||
v22.4.0 template, the same one frank-medium uses). Without this flag the
|
||
binary falls back to the built-in `peg-native` template, which produces a
|
||
~73,600-token prompt for a ~250-token request (280x the expected size).
|
||
With the sharp template wired in, the same request produces a 62–175-token
|
||
prompt. The `Chat format: peg-native` log line is the format-type label,
|
||
not the template indicator — the sharp template is active even when that
|
||
line appears.
|
||
|
||
## Memory budget
|
||
|
||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
|
||
"dedicated" is not the budget). The 27B dense model (26 GiB total) is
|
||
mmap'd in with `-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV +
|
||
buffers), leaving ~84 GiB of aperture headroom. The 8 GiB prompt-cache
|
||
budget (`CACHE_RAM=8192`) is the Kairic Edge qualified config. The model
|
||
is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet `CACHE_RAM`
|
||
floor does not apply.
|
||
|
||
## Payload
|
||
|
||
`scripts/download-frank-large-kairic.sh` (resumable, each file verified
|
||
against its declared byte count + sha256; `--verify-only` re-checks):
|
||
|
||
| file | size | sha256 |
|
||
|---|---|---|
|
||
| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 16,617,792,672 | `360caf73…` |
|
||
| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 8,576,856,064 | `adcbb90a…` |
|
||
| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 2,019,569,664 | `82f93131…` |
|
||
| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 756,953,088 | `3b07e7b1…` |
|
||
|
||
## Client contract
|
||
|
||
- **Port**: 7770 (the same port and alias the unsloth predecessor served).
|
||
- **Alias**: `frank-large`.
|
||
- **Context**: 262,144 (the model's native window), 1 slot.
|
||
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
|
||
--reasoning-budget -1`), extracted to `reasoning_content`.
|
||
- **Vision**: text-only (the model has no vision projector). The fleet's
|
||
image path is [[frank-small-gemma4-12b]] either way.
|
||
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
|
||
{xhigh, medium, low} — the harness folds its levels onto that set in
|
||
`models.yml` (`reasoningEffortMap`).
|
||
|
||
## Measured performance
|
||
|
||
10-question conversation (2026-09-21, post-deploy): **23.99s total, 530
|
||
tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
|
||
MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
|
||
`ngram-mod` and `draft-mtp` active in the journal).
|
||
|
||
**Template verification** (2026-09-21): a 250-token request produces a
|
||
62–175-token prompt (not the 73,600-token prompt the built-in
|
||
`peg-native` template produces). Draft acceptance 0.79 (mean 4.14), the
|
||
model answers correctly, and reasoning lands in `reasoning_content`.
|
||
|
||
## Failure modes worth knowing
|
||
|
||
- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes
|
||
it, and the next build fails. Re-run the build script (it re-checks-out
|
||
the CK commit and re-applies the patch).
|
||
- **Fast-greedy mode rejects the harness**: if you flip to
|
||
`COMPATIBILITY_MODE=0`, the harness's sampling/tools/reasoning-budget
|
||
requests are rejected. Re-test chat before production.
|
||
- **The `charlie12345` fork is NOT the Kairic Edge repo**: the local
|
||
clone at `/home/dank/GitPackages/ROCmFPX` is the `charlie12345/ROCmFPX`
|
||
fork (no kairic branch). The Kairic Edge build uses the `ciru` remote
|
||
(`https://github.com/ciru-ai/ROCmFPX.git`), branch
|
||
`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`, in the worktree at
|
||
`/home/dank/GitPackages/ROCmFPX-kairic/`.
|
||
|
||
## Rollback
|
||
|
||
[[frank-large-unsloth-qwen38-flash-next]] — the unsloth prebuilt payload
|
||
and runtime are intact on disk.
|