frank-large: swap to Kairic Edge 27B IU4 (W4A4) on Strix Halo

Replace the unsloth llama.cpp prebuilt (b11030, Qwen3.8-Flash-Next
UD-IQ4_XS, 125B MoE) with the Kairic Edge runtime (ciru-ai/ROCmFPX,
branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving
jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4).

- Runtime: built from source (scripts/build-kairic-edge-gfx1151.sh),
  TheRock 7.15.0a20260718 toolchain, CK fdf4bb7f + gfx1151-iu4 patch.
- Shape: one stream, full 262144 context (the model's native window),
  1 slot, 8 GiB prompt-cache budget.
- Spec method: native MTP depth 4 + n-gram drafting (24/64/64). DFlash
  is NOT available for this model (no DRAFT_DFLASH spec type).
- Compatibility mode: the harness needs sampling, tools and a reasoning
  budget, which fast-greedy rejects. COMPATIBILITY_MODE=1 enables them.
- Reasoning: enabled (auto + deepseek format, thoughts land in
  reasoning_content).
- Payload: 4 files (15.5 GiB GGUF + 7.99 GiB FFN + 1.88 GiB GDN + 0.70
  GiB GDN-Output), sha256-verified (scripts/download-frank-large-kairic.sh).
This commit is contained in:
Frank committed 2026-09-21 18:56:35 -07:00
1 parent 2f7aadab88
commit b2a74004df
7 files changed
+410 -155

No files matched your search

+25 -22
View File
@@ -1,6 +1,6 @@
# frank-large — unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm
# gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Strix
# Halo iGPU.
# frank-large — Kairic Edge runtime (ciru-ai/ROCmFPX, branch
# kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving
# jcbtc/Qwen3.8-27B-IU4-Kairic-Edge on the Strix Halo iGPU (gfx1151).
#
# These are the tunables read by scripts/serve-frank-large.sh. The flag
# list is owned by the serve script; this file only sets fleet values.
@@ -9,31 +9,34 @@
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
# exception because the serve script itself consumes it for --host.
# The assembled payload (scripts/download-frank-large-unsloth.sh): the
# serve script refuses to start without the main GGUF and the MTP head.
MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS
# The assembled payload (scripts/download-frank-large-kairic.sh): the
# serve script refuses to start without the main GGUF and the three
# sidecars (FFN, GDN, GDN-Output).
MODEL_DIR=/home/dank/models/Qwen3.8-27B-IU4-Kairic-Edge
# The prebuilt runtime (see the serve script for the fetch line).
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
# The Kairic Edge runtime (scripts/build-kairic-edge-gfx1151.sh).
RUNTIME_DIR=/home/dank/GitPackages/ROCmFPX-kairic/build-kairic
# Fleet endpoint: the same port and alias the CIRU predecessor served, so
# Fleet endpoint: the same port and alias the unsloth predecessor served, so
# no client, probe or models.yml entry changes address.
HOST=0.0.0.0
PORT=7770
# One stream, 200k context (operator's call, 2026-09-21). The model's
# trained window is 262144; 200000 is the requested serving shape.
CONTEXT_SIZE=200000
# One stream, full 262144 context (the model's native window).
CONTEXT_SIZE=262144
PARALLEL_SLOTS=1
# MTP shared-Q8_0 head: on. Draft depth 2 is the model card's default
# ("higher drafts more but each guess is accepted less often"). The shared
# head borrows the token embedding and output projection from the main
# model, so the file is 2.6 GB rather than 4.1.
ENABLE_MTP=1
SPEC_DRAFT_N_MAX=2
# Native MTP depth 4 (the model's MTP layers) + n-gram drafting (24/64/64).
# DFlash is NOT available for this model (no DRAFT_DFLASH spec type in the
# Kairic Edge runtime).
SPEC_DRAFT_N_MAX=4
# Text-only for now: the F16 projector is on disk at vision/mmproj-F16.gguf;
# flip to 1 to enable it. The fleet's image path is frank-small either way,
# so models.yml stays input: [text] until this changes.
ENABLE_VISION=0
# The harness needs sampling, tools and a reasoning budget, which the
# fast-greedy path rejects. COMPATIBILITY_MODE=1 enables them (9.7% slower
# per the build guide).
COMPATIBILITY_MODE=1
# Prompt-cache budget: 8 GiB (the Kairic Edge qualified config). The model
# is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet floor does
# not apply.
CACHE_RAM=8192