feat(frank-large): unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (200k/1 slot, MTP depth 2)
Replace the CIRU v4.4.1 Orca runtime with the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB, three HF split parts concatenated) on the Strix Halo iGPU. One conversation, 200,000 ctx, PARALLEL_SLOTS=1, MTP depth 2 via the explicit -md shared-Q8_0 head (sidecar auto-discovery does not search the mtp/ subfolder), text-only. Fleet contract unchanged: port 7770, alias frank-large. - serve-frank-large.sh: unsloth binary, -m/-md, --spec-type draft-mtp --spec-draft-n-max 2, --jinja, preserve_thinking; gates on binary, GGUF, MTP head; conditional --mmproj on ENABLE_VISION=1 - download-frank-large-unsloth.sh: resumable fetch of 3 parts + MTP head + mmproj, each verified against its LFS sha256 fetched live from the Hub tree API; concatenates parts, records/verifies SHA256SUMS - deploy-remote.sh: LLM_LARGE_MODEL/LLM_LARGE_DRAFT paths, pkill pattern on the model path (not the binary name), wait_port_free gate - tests: seams for 200k/1-slot, explicit -md, text-only, handler gates, sweep safety
This commit is contained in:
1 parent
e5a1f8e767
commit
b149d02187
6 files changed
+272
-157
No files matched your search
+26
-31
@@ -1,44 +1,39 @@
|
||||
# frank-large — CIRU v4.4.1 (QSA window hotfix) serving
|
||||
# jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Strix Halo iGPU (gfx1151).
|
||||
# frank-large — unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm
|
||||
# gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Strix
|
||||
# Halo iGPU.
|
||||
#
|
||||
# These are the launcher tunables read by the runtime's
|
||||
# scripts/ciru/run-orca-server.sh (see the serve script). The 36-variable
|
||||
# kernel bundle and the PLE pager defaults live in the runtime's own
|
||||
# profiles/strix-halo-production.env, which the launcher sources AFTER this
|
||||
# file; every profile line is a ${VAR:-default}, so a value set here wins.
|
||||
# These are the tunables read by scripts/serve-frank-large.sh. The flag
|
||||
# list is owned by the serve script; this file only sets fleet values.
|
||||
#
|
||||
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP runtime
|
||||
# reads those bare (the 2026-08-13 TEMP incident). HOST is the one exception
|
||||
# because the launcher itself consumes it for --host; its value doubles as a
|
||||
# JIT-cache directory name, which is why 0.0.0.0 is safe.
|
||||
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
||||
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
|
||||
# exception because the serve script itself consumes it for --host.
|
||||
|
||||
# The assembled payload (scripts/download-frank-large-orca.sh): the launcher
|
||||
# refuses to start without the main GGUF, the MTP head and the PLE sidecar.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca
|
||||
# The assembled payload (scripts/download-frank-large-unsloth.sh): the
|
||||
# serve script refuses to start without the main GGUF and the MTP head.
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS
|
||||
|
||||
# Fleet endpoint: the same port and alias the halogen predecessor served, so
|
||||
# The prebuilt runtime (see the serve script for the fetch line).
|
||||
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
||||
|
||||
# Fleet endpoint: the same port and alias the CIRU predecessor served, so
|
||||
# no client, probe or models.yml entry changes address.
|
||||
HOST=0.0.0.0
|
||||
PORT=7770
|
||||
|
||||
# One stream, full context (operator call, 2026-09-19). MTP requires exactly
|
||||
# one slot in this runtime, so the drafter's constraint and the operator's
|
||||
# ask are the same number. 262144 is the model's trained window; the GGUF
|
||||
# carries no rope scaling.
|
||||
CONTEXT_SIZE=262144
|
||||
# One stream, 200k context (operator's call, 2026-09-21). The model's
|
||||
# trained window is 262144; 200000 is the requested serving shape.
|
||||
CONTEXT_SIZE=200000
|
||||
PARALLEL_SLOTS=1
|
||||
|
||||
# MTP head: on. Depth 4 is the Orca variant's own default (mtp_depth_default
|
||||
# in the launcher); the sidecar is the Q8_0 MTP file from the model repo.
|
||||
# MTP shared-Q8_0 head: on. Draft depth 2 is the model card's default
|
||||
# ("higher drafts more but each guess is accepted less often"). The shared
|
||||
# head borrows the token embedding and output projection from the main
|
||||
# model, so the file is 2.6 GB rather than 4.1.
|
||||
ENABLE_MTP=1
|
||||
SPEC_DRAFT_N_MAX=2
|
||||
|
||||
# Text-only for now: the F16 projector is on disk at
|
||||
# vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj; flip to 1 to enable it. The
|
||||
# fleet's image path is frank-small either way, so models.yml stays
|
||||
# input: [text] until this changes.
|
||||
# Text-only for now: the F16 projector is on disk at vision/mmproj-F16.gguf;
|
||||
# flip to 1 to enable it. The fleet's image path is frank-small either way,
|
||||
# so models.yml stays input: [text] until this changes.
|
||||
ENABLE_VISION=0
|
||||
|
||||
# PLE pager: the 52.4 GiB payload is mmap'd and paged by the engine's own
|
||||
# workers; this is the hot window kept resident. The vendor's production
|
||||
# profile default is 4096.
|
||||
PLE_CACHE_MIB=4096
|
||||
Reference in new issue
Block a user