Frank
b2a74004df
Replace the unsloth llama.cpp prebuilt (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE) with the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4). - Runtime: built from source (scripts/build-kairic-edge-gfx1151.sh), TheRock 7.15.0a20260718 toolchain, CK fdf4bb7f + gfx1151-iu4 patch. - Shape: one stream, full 262144 context (the model's native window), 1 slot, 8 GiB prompt-cache budget. - Spec method: native MTP depth 4 + n-gram drafting (24/64/64). DFlash is NOT available for this model (no DRAFT_DFLASH spec type). - Compatibility mode: the harness needs sampling, tools and a reasoning budget, which fast-greedy rejects. COMPATIBILITY_MODE=1 enables them. - Reasoning: enabled (auto + deepseek format, thoughts land in reasoning_content). - Payload: 4 files (15.5 GiB GGUF + 7.99 GiB FFN + 1.88 GiB GDN + 0.70 GiB GDN-Output), sha256-verified (scripts/download-frank-large-kairic.sh).
43 lines
1.7 KiB
Bash
43 lines
1.7 KiB
Bash
# frank-large — Kairic Edge runtime (ciru-ai/ROCmFPX, branch
|
|
# kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving
|
|
# jcbtc/Qwen3.8-27B-IU4-Kairic-Edge on the Strix Halo iGPU (gfx1151).
|
|
#
|
|
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
|
# list is owned by the serve script; this file only sets fleet values.
|
|
#
|
|
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
|
# exception because the serve script itself consumes it for --host.
|
|
|
|
# The assembled payload (scripts/download-frank-large-kairic.sh): the
|
|
# serve script refuses to start without the main GGUF and the three
|
|
# sidecars (FFN, GDN, GDN-Output).
|
|
MODEL_DIR=/home/dank/models/Qwen3.8-27B-IU4-Kairic-Edge
|
|
|
|
# The Kairic Edge runtime (scripts/build-kairic-edge-gfx1151.sh).
|
|
RUNTIME_DIR=/home/dank/GitPackages/ROCmFPX-kairic/build-kairic
|
|
|
|
# Fleet endpoint: the same port and alias the unsloth predecessor served, so
|
|
# no client, probe or models.yml entry changes address.
|
|
HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# One stream, full 262144 context (the model's native window).
|
|
CONTEXT_SIZE=262144
|
|
PARALLEL_SLOTS=1
|
|
|
|
# Native MTP depth 4 (the model's MTP layers) + n-gram drafting (24/64/64).
|
|
# DFlash is NOT available for this model (no DRAFT_DFLASH spec type in the
|
|
# Kairic Edge runtime).
|
|
SPEC_DRAFT_N_MAX=4
|
|
|
|
# The harness needs sampling, tools and a reasoning budget, which the
|
|
# fast-greedy path rejects. COMPATIBILITY_MODE=1 enables them (9.7% slower
|
|
# per the build guide).
|
|
COMPATIBILITY_MODE=1
|
|
|
|
# Prompt-cache budget: 8 GiB (the Kairic Edge qualified config). The model
|
|
# is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet floor does
|
|
# not apply.
|
|
CACHE_RAM=8192
|