frank-large: ROCmFPX llama.cpp on the Q8_0_ROCMFPX_AGENT file, MTP drafting
The repo's highest-quality rung is ftype 115: 311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a type only the ROCmFPX fork reads, so the runtime moves from gufo to scripts/build-rocmfpx.sh's pinned fork build (36e9acd) — the first build on this host measured to load this file without the 2026-08-25 set_rows segfault. One session at the native 262,144 window, f16 KV (64 KiB/token, 16 GiB), the repo's Q4_0 MTP head drafting 4 tokens, plus the bundled BF16 projector so the endpoint has vision again. The serve script now honors caller environment overrides over the env file (a plain source let the file win, which made single-run overrides no-ops) and defaults apply only when a value is unset, so an explicit LLAMA_BIN='' is the error it reads as.
This commit is contained in:
Frank committed
1 parent
d82c5c03b7
commit
125d04967f
6 files changed
+608
-495
No files matched your search
+33
-33
@@ -1077,27 +1077,29 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large is gufo (gufo-org/gufo), the Strix Halo inference engine built
|
||||
# from the pinned v0.1.1 tag by scripts/build-gufo.sh, serving
|
||||
# unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the DFlash2 drafter
|
||||
# (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Strix Halo iGPU. It is a
|
||||
# ROOT unit: the runtime is a native binary linked against the host ROCm,
|
||||
# so there is no container image to install and no generated unit to dodge.
|
||||
# relink_unit points /etc/systemd/system/frank-large.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
# frank-large is the ROCmFPX fork of llama.cpp (kingjones30/ROCmFPX, pinned by
|
||||
# scripts/build-rocmfpx.sh), serving
|
||||
# kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF:Q8_0_ROCMFPX_AGENT with the
|
||||
# repo's own Q4_0 MTP draft head and its BF16 vision projector, on the Strix
|
||||
# Halo iGPU. It is a ROOT unit: the runtime is a native binary linked against
|
||||
# the host ROCm, so there is no container image to install and no generated
|
||||
# unit to dodge. relink_unit points /etc/systemd/system/frank-large.service at
|
||||
# the repo file and boot persistence is that unit's [Install] section.
|
||||
#
|
||||
# The runtime gate matters as much as the payload gate: gufo is built from
|
||||
# source on this host (there is no prebuilt release), so a machine whose
|
||||
# The runtime gate matters as much as the payload gate: the fork is built from
|
||||
# source on this host, and the payload is ftype 115 (tensor type 103,
|
||||
# GGML_TYPE_Q8_0_ROCMFPX) which only this fork can read — so a machine whose
|
||||
# /home/dank/GitPackages tree was never built has a unit that can never
|
||||
# become healthy, and a green deploy would leave the endpoint dark.
|
||||
#
|
||||
# It replaces the stock llama.cpp prebuilt (unsloth b11030), which had
|
||||
# replaced the Kairic Edge runtime (ciru-ai/ROCmFPX). The port gate is the
|
||||
# same as every cutover since 2026-09-16: nothing starts until 7770 is
|
||||
# released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
# It replaces gufo v0.1.1 + unsloth UD-Q4_K_XL (the rollback target, still
|
||||
# built by scripts/build-gufo.sh with its payload on disk), which had replaced
|
||||
# the stock llama.cpp prebuilt. The port gate is the same as every cutover
|
||||
# since 2026-09-16: nothing starts until 7770 is released.
|
||||
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
|
||||
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
|
||||
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
LLM_LARGE_MMPROJ="${LLM_LARGE_MMPROJ:-/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
|
||||
retire_halogen_large_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
||||
@@ -1132,17 +1134,19 @@ wait_port_free() {
|
||||
deploy_llm_large() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
# endpoint dark with nothing in the deploy log to say why. The gates
|
||||
# mirror serve-frank-large.sh's own predicates exactly: runtime
|
||||
# executable, target GGUF, MTP head, mmproj.
|
||||
if [[ ! -x "$LLM_LARGE_RUNTIME" ]]; then
|
||||
echo "ERROR: gufo runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-gufo.sh" >&2
|
||||
echo "ERROR: ROCmFPX runtime absent at $LLM_LARGE_RUNTIME" >&2
|
||||
echo "frank-large: build it: bash scripts/build-rocmfpx.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
local missing=0 f
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT"; do
|
||||
for f in "$LLM_LARGE_MODEL" "$LLM_LARGE_DRAFT" "$LLM_LARGE_MMPROJ"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "ERROR: payload absent at $f" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large.sh" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-kingjones.sh" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
@@ -1152,18 +1156,14 @@ deploy_llm_large() {
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# Then sweep a stray manual run -- that is how a bench harness leaves the
|
||||
# port busy. Match the DRAFTER path, which every frank-large command
|
||||
# line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
|
||||
# payload swap: the target moved to a different repository while the
|
||||
# drafter stayed put, so keying on the target filename would have quietly
|
||||
# stopped matching. It is also the right key in principle — it names this
|
||||
# endpoint's speculative drafter, not a binary. Match that, not a binary
|
||||
# name: frank-small on this host serves a different GGUF over llama-server
|
||||
# with no DFlash2 drafter and must not be killed. The pattern never
|
||||
# appears in the deploying shell's own command line, so -f is safe here
|
||||
# (it is not in `bash deploy-remote.sh`).
|
||||
# port busy. Match the TARGET GGUF name, which every frank-large command
|
||||
# line carries via --model and which no other process on this host uses:
|
||||
# frank-small also runs llama-server here (with the MiMo GGUF) and must
|
||||
# not be killed, so keying on the binary name would take the 3090 endpoint
|
||||
# down with it. The pattern never appears in the deploying shell's own
|
||||
# command line, so -f is safe here (it is not in `bash deploy-remote.sh`).
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
|
||||
pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT' 2>/dev/null || true
|
||||
# Nothing may start until the predecessor has released the port. On
|
||||
# 2026-09-16 the cutover failed exactly here: the old server logged
|
||||
# "Deactivated successfully", the next engine tried to bind 7770 a second
|
||||
|
||||
Reference in new issue
Block a user