Frank
b149d02187
Replace the CIRU v4.4.1 Orca runtime with the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB, three HF split parts concatenated) on the Strix Halo iGPU. One conversation, 200,000 ctx, PARALLEL_SLOTS=1, MTP depth 2 via the explicit -md shared-Q8_0 head (sidecar auto-discovery does not search the mtp/ subfolder), text-only. Fleet contract unchanged: port 7770, alias frank-large. - serve-frank-large.sh: unsloth binary, -m/-md, --spec-type draft-mtp --spec-draft-n-max 2, --jinja, preserve_thinking; gates on binary, GGUF, MTP head; conditional --mmproj on ENABLE_VISION=1 - download-frank-large-unsloth.sh: resumable fetch of 3 parts + MTP head + mmproj, each verified against its LFS sha256 fetched live from the Hub tree API; concatenates parts, records/verifies SHA256SUMS - deploy-remote.sh: LLM_LARGE_MODEL/LLM_LARGE_DRAFT paths, pkill pattern on the model path (not the binary name), wait_port_free gate - tests: seams for 200k/1-slot, explicit -md, text-only, handler gates, sweep safety
40 lines
1.7 KiB
Bash
40 lines
1.7 KiB
Bash
# frank-large — unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm
|
|
# gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Strix
|
|
# Halo iGPU.
|
|
#
|
|
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
|
# list is owned by the serve script; this file only sets fleet values.
|
|
#
|
|
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
|
# exception because the serve script itself consumes it for --host.
|
|
|
|
# The assembled payload (scripts/download-frank-large-unsloth.sh): the
|
|
# serve script refuses to start without the main GGUF and the MTP head.
|
|
MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS
|
|
|
|
# The prebuilt runtime (see the serve script for the fetch line).
|
|
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
|
|
|
# Fleet endpoint: the same port and alias the CIRU predecessor served, so
|
|
# no client, probe or models.yml entry changes address.
|
|
HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# One stream, 200k context (operator's call, 2026-09-21). The model's
|
|
# trained window is 262144; 200000 is the requested serving shape.
|
|
CONTEXT_SIZE=200000
|
|
PARALLEL_SLOTS=1
|
|
|
|
# MTP shared-Q8_0 head: on. Draft depth 2 is the model card's default
|
|
# ("higher drafts more but each guess is accepted less often"). The shared
|
|
# head borrows the token embedding and output projection from the main
|
|
# model, so the file is 2.6 GB rather than 4.1.
|
|
ENABLE_MTP=1
|
|
SPEC_DRAFT_N_MAX=2
|
|
|
|
# Text-only for now: the F16 projector is on disk at vision/mmproj-F16.gguf;
|
|
# flip to 1 to enable it. The fleet's image path is frank-small either way,
|
|
# so models.yml stays input: [text] until this changes.
|
|
ENABLE_VISION=0
|