frank-large: swap to stock llama.cpp serving Qwen3.8-27B UD-Q8_K_XL with DFlash2

Replace the Kairic Edge ROCmFPX build with the unsloth b11030 stock llama.cpp prebuilt (ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF: UD-Q8_K_XL (a 27B dense model, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU. Serving shape: one 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, greedy sampling, Qwen-Sharp v22.4.0 template. - scripts/serve-frank-large.sh: stock llama.cpp + DFlash2 spec decode - scripts/frank-large.env: new model/drafter/runtime, 262144 ctx / 1 slot - scripts/download-frank-large.sh: per-file fetch, verify mode, idempotent - scripts/frank-large.service: updated description - scripts/deploy-remote.sh: updated pkill pattern + asset gates - tests/frank_large_unit.bats: new model/runtime/spec assertions - docs: new runbook (active), Kairic Edge runbook retired (rollback target) Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-21 21:50:36 -07:00
1 parent 43b8258061
commit 7689092db8
@@ -1,34 +1,32 @@
 #!/usr/bin/env bash
-# Serve jcbtc/Qwen3.8-27B-IU4-Kairic-Edge as frank-large on
-# Dankclaw's Strix Halo iGPU (gfx1151), using the Kairic Edge runtime
-# (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) —
-# the qualified IU4 (W4A4) serving path with the FFN, GDN and GDN-Output
-# sidecars.
+# Serve unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL as frank-large on
+# Dankclaw's Strix Halo iGPU (gfx1151), using stock llama.cpp
+# (mainline, the unsloth b11030 prebuilt).
 #
-# Runtime: built from source (scripts/build-kairic-edge-gfx1151.sh), output at
-# /home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server. The
-# Kairic Edge path is a llama.cpp build with the PromptForge sidecar env
-# block (the certified runner's exact env + flags).
+# Runtime: /home/dank/GitPackages/unsloth-llama-b11030/llama-server
+# (mainline, ROCm gfx1151 prebuilt). The build supports the qwen3_5
+# architecture and the DFlash2 drafter (llama_model_qwen35,
+# llama_model_dflash, common_speculative_impl_draft_dflash).
 #
 # Shape: one stream, full 262144 context (the model's native window), all
-# layers on the iGPU (-ngl 999). The 27B dense model (15.5 GiB GGUF + 7.99
-# GiB FFN sidecar + 1.88 GiB GDN sidecar + 0.70 GiB GDN-Output sidecar)
-# fits the iGPU's GTT aperture (~114 GiB).
+# layers on the iGPU (--n-gpu-layers 999). The 27B dense model (31.46 GiB GGUF) + the
+# DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) + compute buffers fits the
+# iGPU's GTT aperture (~114 GiB).
 #
-# Spec method: native MTP depth 4 (the model's MTP layers) + n-gram drafting
-# (24/64/64). DFlash is NOT available for this model (no DRAFT_DFLASH spec
-# type in the Kairic Edge runtime).
+# Spec method: DFlash2 (a block-diffusion drafter) at depth 4. The drafter
+# has a sliding window of 2048 (small draft KV, ~40 MiB).
 #
-# Compatibility mode: the harness needs sampling, tools and a reasoning
-# budget, which the fast-greedy path rejects. COMPATIBILITY_MODE=1 enables
-# them (9.7% slower per the build guide). Reasoning is enabled (auto +
-# deepseek format, so thoughts land in reasoning_content).
+# KV: q8_0 quantized (--cache-type-k q8_0 --cache-type-v q8_0). The unsloth b11030
+# build does not support fp8 KV (allowed: f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/
+# q5_1); q8_0 is the highest-precision 8-bit option and preserves the ~8 GiB footprint.
+#
+# Text-only (no mmproj) — matching the prior frank-large.
 #
 # The Qwen3.8 family template keeps prior thinking blocks across turns when
 # preserve_thinking is set; pass --jinja and the kwarg so multi-turn agents
-# do not lose the trail. A request-level chat_template_kwargs still wins.
-# The template gates reasoning_effort to {xhigh, medium, low} — the harness
-# folds its levels onto that set in models.yml (reasoningEffortMap).
+# do not lose the trail. The template gates reasoning_effort to {xhigh,
+# medium, low} — the harness folds its levels onto that set in models.yml
+# (reasoningEffortMap).
 #
 # Tunables come from frank-large.env unless FRANK_LARGE_ENV_FILE overrides
 # it; anything not set there falls through to the defaults below.
@@ -37,96 +35,72 @@ set -euo pipefail
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
 if [[ -f "$ENV_FILE" ]]; then
-    # shellcheck disable=SC1090
+    set -a
    source "$ENV_FILE"
+    set +a
 fi

 # --- defaults (used when the env file does not set them) ------------------
-: "${RUNTIME_DIR:=/home/dank/GitPackages/ROCmFPX-kairic/build-kairic}"
-: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-IU4-Kairic-Edge}"
+: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
+: "${MODEL_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
 : "${HOST:=0.0.0.0}"
 : "${PORT:=7770}"
 : "${CONTEXT_SIZE:=262144}"
 : "${PARALLEL_SLOTS:=1}"
 : "${SPEC_DRAFT_N_MAX:=4}"
-: "${COMPATIBILITY_MODE:=1}"
+: "${CACHE_RAM:=8192}"
 TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/../dankserver/frank-medium/qwen-sharp-v22.4.0.jinja}"

-SERVER_BIN="$RUNTIME_DIR/bin/llama-server"
-MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-IU4-Kairic-Edge.gguf"
-FFN_SIDECAR="$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-FFN.pfs"
-GDN_SIDECAR="$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN.pfs"
-GDN_OUTPUT_SIDECAR="$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs"
+SERVER_BIN="$RUNTIME_DIR/llama-server"
+MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-UD-Q8_K_XL.gguf"
+DRAFTER_FILE="$MODEL_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf"

 if [[ ! -x "$SERVER_BIN" ]]; then
-    echo "ERROR: Kairic Edge runtime not found at $SERVER_BIN" >&2
-    echo "       Build it with: scripts/build-kairic-edge-gfx1151.sh" >&2
+    echo "runtime not found at $SERVER_BIN" >&2
    exit 1
 fi
-for f in "$MODEL_FILE" "$FFN_SIDECAR" "$GDN_SIDECAR" "$GDN_OUTPUT_SIDECAR"; do
-    if [[ ! -r "$f" ]]; then
-        echo "ERROR: missing asset: $f" >&2
-        echo "       Fetch it with: scripts/download-frank-large-kairic.sh" >&2
+for f in "$MODEL_FILE" "$DRAFTER_FILE"; do
+    if [[ ! -f "$f" ]]; then
+        echo "missing asset $f (run scripts/download-frank-large.sh)" >&2
        exit 1
    fi
 done

-# --- PromptForge env block (the Kairic Edge runner's exact env) ----------
+# --- env (the unsloth build's ROCm setup) --------------------------------
 export HIP_VISIBLE_DEVICES=0
 export HSA_OVERRIDE_GFX_VERSION=11.5.1
 export LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/llvm/lib:${LD_LIBRARY_PATH:-}"
-export GGML_CUDA_GRAPH_OPT=0
-# Fast-greedy mode (0) rejects sampling/tools/a reasoning budget; compatibility
-# mode (1) enables them. The harness needs all three, so default to 1.
-if [[ "$COMPATIBILITY_MODE" == "1" ]]; then
-    export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0
-    export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0
-else
-    export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1
-    export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=1
-fi
-export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
-export PROMPTFORGE_TARGET_ONLY=0
-export PROMPTFORGE_MODE=iu4_ffn
-export PROMPTFORGE_IU4_SIDECAR="$FFN_SIDECAR"
-export PROMPTFORGE_SIDECAR="$FFN_SIDECAR"
-export PROMPTFORGE_IU4_HADAMARD=1
-export PROMPTFORGE_IU4_SEGMENTED=0
-export PROMPTFORGE_ENABLE_FFN_KEEPERS=0
-export PROMPTFORGE_FFN_KEEPERS=late6
-export PROMPTFORGE_ENABLE_GDN=1
-export PROMPTFORGE_ENABLE_GDN_W8=0
-export PROMPTFORGE_GDN_SIDECAR_OVERRIDE=1
-export PROMPTFORGE_GDN_SIDECAR="$GDN_SIDECAR"
-export PROMPTFORGE_GDN_IU4_HADAMARD=1
-export PROMPTFORGE_ENABLE_GDN_KEEPERS=0
-export PROMPTFORGE_ENABLE_GDN_OUTPUT=1
-export PROMPTFORGE_GDN_OUTPUT_SIDECAR="$GDN_OUTPUT_SIDECAR"
-export PROMPTFORGE_GDN_OUTPUT_KEEPERS=v3_lateq6
-export PROMPTFORGE_ENABLE_SMALLM_IU4=1
-export PROMPTFORGE_ENABLE_SMALLM_GDN_IU4=1
-export PROMPTFORGE_ENABLE_SMALLM_GDN_OUTPUT_IU4=0
-export PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0

-# --- the serve command (the Kairic Edge runner's flags, fleet-adapted) ----
+# --- the serve command ---------------------------------------------------
 ARGS=(
-    -m "$MODEL_FILE" --alias frank-large
-    --host "$HOST" --port "$PORT" --jinja
-    --chat-template-kwargs '{"preserve_thinking":true}'
+    --host "$HOST"
+    --port "$PORT"
+    --model "$MODEL_FILE"
+    --alias frank-large
+    --ctx-size "$CONTEXT_SIZE"
+    --parallel "$PARALLEL_SLOTS"
+    --n-gpu-layers 999
+    --jinja
    --chat-template-file "$TEMPLATE_FILE"
-    -dev ROCm0 -ngl 999
-    -c "$CONTEXT_SIZE" -b 2048 -ub 512 -fa on -ctk f16 -ctv f16
-    -t 16 -tb 32 -np "$PARALLEL_SLOTS" -ctxcp 32 --cache-ram "$CACHE_RAM"
-    --cache-prompt --cache-idle-slots --metrics
-    --kairic-edge
-    --spec-type draft-mtp
-    --spec-draft-device ROCm0 --spec-draft-ngl 999
-    --spec-draft-type-k f16 --spec-draft-type-v f16
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" --spec-draft-n-min 0
-    --spec-draft-p-min 0.0 --spec-draft-p-split 0.10
-    --spec-draft-backend-sampling
-    --temp 0 --top-p 1.0 --top-k 0 --min-p 0.0
-    --reasoning auto --reasoning-format deepseek --reasoning-budget -1
+    --chat-template-kwargs '{"preserve_thinking":true}'
+    --cache-type-k q8_0
+    --cache-type-v q8_0
+    --cache-ram "$CACHE_RAM"
+    # DFlash2 spec decode (a block-diffusion drafter)
+    --spec-type draft-dflash
+    --model-draft "$DRAFTER_FILE"
+    --spec-draft-n-max "$SPEC_DRAFT_N_MAX"
+    --spec-draft-n-min 0
+    --spec-draft-device ROCm0
+    # Greedy sampling (the DFlash2 drafter needs greedy verification)
+    --temp 0
+    --top-p 1.0
+    --top-k 0
+    --min-p 0.0
+    # Reasoning (thoughts land in reasoning_content)
+    --reasoning auto
+    --reasoning-format deepseek
+    --reasoning-budget -1
 )

 exec "$SERVER_BIN" "${ARGS[@]}"