dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
24f7a7772d7f222654a463de166b98791cb795cc
serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-kairic-edge-qwen38-27b.md
T
Frank 97f60b35ef docs: document Qwen-Sharp template wiring and 73k-token diagnosis 
The serve script now passes --chat-template-file pointing at the
Qwen-Sharp v22.4.0 template. Without it the binary falls back to the
built-in peg-native template, which produces a ~73,600-token prompt for
a ~250-token request. With the sharp template wired in, the same request
produces a 62-175-token prompt.
2026-09-21 20:11:00 -07:00

9.4 KiB
Raw Blame History

type: reference title: Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large) description: The active frank-large runbook — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing. tags:

  • llm
  • kairic-edge
  • rocmfpx
  • dankclaw
  • frank-large
  • strix-halo
  • gfx1151
  • iu4
  • w4a4
  • rocm
  • mtp
  • promptforge timestamp: 2026-09-21

Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)

ACTIVE since 2026-09-21. It replaced the unsloth llama.cpp prebuilt (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE — see frank-large-unsloth-qwen38-flash-next, the current rollback target), which had replaced the CIRU v4.4.1 Orca runtime (see frank-large-ciru-orca). Deployed as a root systemd unit (scripts/frank-large.service, User=dank), port 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to frank-small-gemma4-12b.

The model is jcbtc/Qwen3.8-27B-IU4-Kairic-Edge — a 27B dense Qwen3.8 at IU4 (W4A4, integer-weights / integer-activations). It is a fundamentally different architecture from the 125B MoE it replaces: no PLE, no MoE experts, no n-gram embedding. The payload is four files: the 15.5 GiB main GGUF plus three .pfs sidecars (FFN 7.99 GiB, GDN 1.88 GiB, GDN-Output 0.70 GiB) — the PromptForge IU4 weight shards the Kairic Edge runtime loads alongside the main GGUF.

The runtime is the Kairic Edge build of ciru-ai/ROCmFPX (branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5), built from source (scripts/build-kairic-edge-gfx1151.sh). It is a llama.cpp build with the PromptForge sidecar env block and the --kairic-edge qualified serving path.

Why this swap

  • The operator asked for the Kairic Edge 27B model at one conversation, full context (262,144) (2026-09-21).
  • The Kairic Edge runtime is the qualified IU4 (W4A4) serving path — no standard llama.cpp quant, no PLE sidecar payload, no QSA hotfix: one GGUF, three sidecars, done.
  • The spec method is native MTP depth 4 + n-gram drafting (24/64/64). DFlash is NOT available for this model (no DRAFT_DFLASH spec type in the Kairic Edge runtime).

Shape

One root unit, one process, one published port:

half port reachable
llama-server (OpenAI front-end + engine, single process) 7770 published as host 7770

Runtime

  • Source build: scripts/build-kairic-edge-gfx1151.sh (TheRock 7.15.0a20260718 toolchain, AMD clang 23.0.0, HIP 7.15.0, GCC 13.3.0, CMake 4.4.2, Ninja 1.13.0). Output at /home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server.
  • Composable Kernel: fdf4bb7fcc984811cef48ce817d89aac064b984a + patches/composable-kernel-gfx1151-iu4.patch (SHA-256 ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5). The CK config.h is generated by CMake's configure_file (line 334 of CK's CMakeLists.txt) — it is untracked, so a git clean removes it.
  • GPU target: gfx1151 (Strix Halo). The build takes ~4 minutes (JOBS=$(nproc)).
  • v1.2 changes: Exact 65-row n-gram verification switched to the compact authoritative path by default; the native IU4 M65 verifier is retained only for diagnostics.

Spec method

  • Native MTP depth 4 (--spec-type draft-mtp --spec-draft-n-max 4): the model's MTP layers draft 4 tokens per step.
  • N-gram drafting (24/64/64): the ngram-mod matcher chains behind the MTP drafter (the --spec-type draft-mtp value builds both).
  • DFlash is NOT available: the Kairic Edge runtime's spec-type constants are DRAFT_EAGLE, DRAFT_MTP, DRAFT_SIMPLE, NGRAM_CACHE, NGRAM_MAP_K, NGRAM_MOD, NGRAM_SIMPLE, NONE — no DRAFT_DFLASH. The z-lab DFlash2-FP8 drafter (frank-medium's drafter) cannot be used.
  • Draft device: --spec-draft-device ROCm0 --spec-draft-ngl 999 (the drafter runs on the iGPU, all layers).

Compatibility mode

The Kairic Edge runtime has two modes:

  • Fast-greedy (KAIRIC_EDGE_COMPATIBILITY_MODE=0, the runner's default): rejects request features that require full target logits, including sampling, penalties, probabilities, grammar-constrained tool calls, logit bias, LoRA, and a reasoning budget.
  • Compatibility (KAIRIC_EDGE_COMPATIBILITY_MODE=1): enables all of the above (9.7% slower per the build guide).

The harness (OMP) needs sampling, tools and a reasoning budget, so COMPATIBILITY_MODE=1 is the deployed shape. In compatibility mode: LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 and PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0.

Known gotcha: the greedy-argmax fastpath (fast mode) rejects non-greedy requests and reasoning budgets. Re-test chat before production if you flip to fast mode.

PromptForge sidecars

The three .pfs sidecars are wired via the PROMPTFORGE_* env block (the certified runner's exact env):

sidecar env var size
FFN PROMPTFORGE_IU4_SIDECAR / PROMPTFORGE_SIDECAR 7.99 GiB
GDN PROMPTFORGE_GDN_SIDECAR 1.88 GiB
GDN-Output PROMPTFORGE_GDN_OUTPUT_SIDECAR 0.70 GiB

The serve script (scripts/serve-frank-large.sh) validates binary → main GGUF → 3 sidecars and exits with a pointer message if any is missing, then execs llama-server with --jinja --alias frank-large --chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE (Qwen3.8's template needs preserve_thinking to carry thinking blocks across turns; a request-level value still wins).

Chat template

The serve script passes --chat-template-file pointing at dankserver/frank-medium/qwen-sharp-v22.4.0.jinja (the Qwen-Sharp v22.4.0 template, the same one frank-medium uses). Without this flag the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request (280x the expected size). With the sharp template wired in, the same request produces a 62–175-token prompt. The Chat format: peg-native log line is the format-type label, not the template indicator — the sharp template is active even when that line appears.

Memory budget

The iGPU addresses 114 GiB (the GTT aperture over the shared LPDDR, gtt_total in /sys/class/drm/card1/device/; rocm-smi's 2 GiB "dedicated" is not the budget). The 27B dense model (26 GiB total) is mmap'd in with -ngl 999; budget at 262k ≈ 30 GiB (weights + KV + buffers), leaving ~84 GiB of aperture headroom. The 8 GiB prompt-cache budget (CACHE_RAM=8192) is the Kairic Edge qualified config. The model is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet CACHE_RAM floor does not apply.

Payload

scripts/download-frank-large-kairic.sh (resumable, each file verified against its declared byte count + sha256; --verify-only re-checks):

file size sha256
Qwen3.8-27B-IU4-Kairic-Edge.gguf 16,617,792,672 360caf73…
Qwen3.8-27B-Kairic-IU4-FFN.pfs 8,576,856,064 adcbb90a…
Qwen3.8-27B-Kairic-IU4-GDN.pfs 2,019,569,664 82f93131…
Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs 756,953,088 3b07e7b1…

Client contract

  • Port: 7770 (the same port and alias the unsloth predecessor served).
  • Alias: frank-large.
  • Context: 262,144 (the model's native window), 1 slot.
  • Reasoning: enabled (--reasoning auto --reasoning-format deepseek --reasoning-budget -1), extracted to reasoning_content.
  • Vision: text-only (the model has no vision projector). The fleet's image path is frank-small-gemma4-12b either way.
  • Reasoning effort: the Qwen3.8 template gates reasoning_effort to {xhigh, medium, low} — the harness folds its levels onto that set in models.yml (reasoningEffortMap).

Measured performance

10-question conversation (2026-09-21, post-deploy): 23.99s total, 530 tokens, 22.1 t/s average. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The MTP depth 4 + n-gram spec method is working (draft acceptance logged, both ngram-mod and draft-mtp active in the journal).

Template verification (2026-09-21): a 250-token request produces a 62–175-token prompt (not the 73,600-token prompt the built-in peg-native template produces). Draft acceptance 0.79 (mean 4.14), the model answers correctly, and reasoning lands in reasoning_content.

Failure modes worth knowing

  • CK config.h is untracked: a git clean in the CK checkout removes it, and the next build fails. Re-run the build script (it re-checks-out the CK commit and re-applies the patch).
  • Fast-greedy mode rejects the harness: if you flip to COMPATIBILITY_MODE=0, the harness's sampling/tools/reasoning-budget requests are rejected. Re-test chat before production.
  • The charlie12345 fork is NOT the Kairic Edge repo: the local clone at /home/dank/GitPackages/ROCmFPX is the charlie12345/ROCmFPX fork (no kairic branch). The Kairic Edge build uses the ciru remote (https://github.com/ciru-ai/ROCmFPX.git), branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, in the worktree at /home/dank/GitPackages/ROCmFPX-kairic/.

Rollback

frank-large-unsloth-qwen38-flash-next — the unsloth prebuilt payload and runtime are intact on disk.

Reference in new issue View Git Blame Copy Permalink