FrankandClaude
7689092db8
Replace the Kairic Edge ROCmFPX build with the unsloth b11030 stock llama.cpp prebuilt (ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF: UD-Q8_K_XL (a 27B dense model, 31.5 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU. Serving shape: one 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, greedy sampling, Qwen-Sharp v22.4.0 template. - scripts/serve-frank-large.sh: stock llama.cpp + DFlash2 spec decode - scripts/frank-large.env: new model/drafter/runtime, 262144 ctx / 1 slot - scripts/download-frank-large.sh: per-file fetch, verify mode, idempotent - scripts/frank-large.service: updated description - scripts/deploy-remote.sh: updated pkill pattern + asset gates - tests/frank_large_unit.bats: new model/runtime/spec assertions - docs: new runbook (active), Kairic Edge runbook retired (rollback target) Co-Authored-By: Claude <noreply@anthropic.com>
37 lines
1.4 KiB
Bash
37 lines
1.4 KiB
Bash
# frank-large — stock llama.cpp (mainline, the unsloth b11030 prebuilt)
|
|
# serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on the Strix Halo iGPU
|
|
# (gfx1151).
|
|
#
|
|
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
|
# list is owned by the serve script; this file only sets fleet values.
|
|
#
|
|
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
|
# exception because the serve script itself consumes it for --host.
|
|
|
|
# The payload (scripts/download-frank-large.sh): the serve script refuses to
|
|
# start without the main GGUF and the DFlash2 drafter.
|
|
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL
|
|
|
|
# The stock llama.cpp runtime (the unsloth b11030 prebuilt).
|
|
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
|
|
|
# Fleet endpoint: the same port and alias the prior deployments served, so
|
|
# no client, probe or models.yml entry changes address.
|
|
HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# One stream, full 262144 context (the model's native window).
|
|
CONTEXT_SIZE=262144
|
|
PARALLEL_SLOTS=1
|
|
|
|
# DFlash2 spec decode (a block-diffusion drafter) at depth 4. The drafter
|
|
# has a sliding window of 2048 (small draft KV, ~40 MiB).
|
|
SPEC_DRAFT_N_MAX=4
|
|
|
|
# Prompt-cache budget: 8 GiB. The model is a 27B hybrid (16 full attention
|
|
# + 48 Gated DeltaNet linear attention); its DeltaNet state is smaller than
|
|
# a primarily-DeltaNet model, so the fleet's 12.8 GiB DeltaNet floor does
|
|
# not apply.
|
|
CACHE_RAM=8192
|