frank-primary: vLLM 0.30 UVA-offload shape (pre-deploy commit)
The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the FP8 cutover on :7770 — same port, same alias, no client moves. Serve shape: locally built ROCm gfx1201 image (localhost/frank-primary:vllm030) execs via podman as dank from the root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload: --offload-backend uva pins the PLE family (47.75 GiB, 138 tensors: 131 n-gram table shards + 7 per-layer conv/proj) + routed experts (114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined, ~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card strands 2.6 GiB of experts on the cards). Parsers per the host's own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM 0.30 emits the channel as `reasoning` — reasoning_content is renamed server-side) + enable_thinking/preserve_thinking template kwargs. Spec decode stays env-wired and OFF at bring-up. Deployer deploy_llm_primary rewritten: image gate, index-driven shard-count gate (131), stale-container cleanup, port wait, and the false-green hardening — health is only green when the unit's own MainPID is live behind the answer (2026-09-26 lesson). Seam tests rewritten to the vLLM contract (15 gates): offload arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite 349/349 green.
This commit is contained in:
Frank
+54
-79
@@ -1183,98 +1183,73 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is unsloth llama.cpp (the b11030-mix
|
||||
# prebuilt, ROCm gfx120X, bundled runtime) serving Qwen3.8-Flash-Next at
|
||||
# Q8_0 on the two R9700s, as a ROOT unit — the same shape as frank-large on
|
||||
# Dankclaw: a host binary, no container image, no generated unit.
|
||||
# relink_unit points /etc/systemd/system/frank-primary.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
# deploy_llm_primary — frank-primary is vLLM 0.30 (locally built ROCm image,
|
||||
# gfx1201) serving Qwen3.8-Flash-Next-FP8 on the two R9700s as a ROOT unit
|
||||
# whose serve script execs a rootless-podman container as dank. Same port and
|
||||
# alias as the llama.cpp endpoint it replaces (2026-09-27 cutover); the
|
||||
# harness models.yml and homepage cards are unchanged.
|
||||
#
|
||||
# The 2026-09-26 swap that replaced the Lucebox DS4.1 Quadlet on this port.
|
||||
# The payload is six Q8_0 split parts plus the shared MTP head and the
|
||||
# projector, so the gate walks every artifact at its declared byte count (a
|
||||
# partial download must never be served — and stat -c %s, never wc -c, which
|
||||
# read the whole 377 GiB lucebox file for minutes on 2026-09-26).
|
||||
LLM_PRIMARY_RUNTIME=/home/dank/GitPackages/unsloth-llama-b11030/llama-server
|
||||
LLM_PRIMARY_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
|
||||
LLM_PRIMARY_PART_SIZES=(
|
||||
10946624
|
||||
682434912
|
||||
54400261312
|
||||
49446841216
|
||||
49668930400
|
||||
34015618784
|
||||
)
|
||||
LLM_PRIMARY_DRAFT="$LLM_PRIMARY_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"
|
||||
LLM_PRIMARY_DRAFT_SIZE=2786568256
|
||||
LLM_PRIMARY_MMPROJ="$LLM_PRIMARY_ROOT/mmproj-BF16.gguf"
|
||||
LLM_PRIMARY_MMPROJ_SIZE=907542944
|
||||
|
||||
# The lucebox quadlet held this port until the 2026-09-26 cutover. Its
|
||||
# generated user unit is host state, not repo state: stop it and move the
|
||||
# installed .container into dated home trash (the same retire pattern as
|
||||
# retire_halogen_large_quadlet). The image and the 377 GiB payload stay on
|
||||
# disk as the rollback target.
|
||||
retire_lucebox_primary_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260926-llamacpp-cutover/quadlet"
|
||||
systemctl --user stop frank-primary.service 2>/dev/null || true
|
||||
if [[ -e "$quadlet_unit_dir/frank-primary.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/frank-primary.container" "$trash/frank-primary.container"
|
||||
echo "retired quadlet frank-primary.container -> $trash"
|
||||
fi
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
# Gates: the image must exist in dank's rootless storage, and the payload
|
||||
# must be complete — every shard named by the model's own
|
||||
# model.safetensors.index.json present on disk (131 shards, 172.78 GiB;
|
||||
# per-shard sha256 + byte-count verification is download-frank-primary-fp8.sh's
|
||||
# job at fetch time, never re-hashed here).
|
||||
LLM_PRIMARY_IMAGE=localhost/frank-primary:vllm030
|
||||
LLM_PRIMARY_ROOT=/home/dank/models/Qwen3.8-Flash-Next-FP8
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-vllm
|
||||
|
||||
deploy_llm_primary() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
local missing=0 i part want_size got_size
|
||||
if [[ ! -x "$LLM_PRIMARY_RUNTIME" ]]; then
|
||||
echo "ERROR: runtime absent at $LLM_PRIMARY_RUNTIME" >&2
|
||||
echo "frank-primary: fetch the unsloth b11030-mix gfx120X prebuilt (see the runbook)" >&2
|
||||
missing=1
|
||||
fi
|
||||
for i in 1 2 3 4 5 6; do
|
||||
part="$LLM_PRIMARY_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-0000$i-of-00006.gguf"
|
||||
want_size="${LLM_PRIMARY_PART_SIZES[$((i-1))]}"
|
||||
got_size="$(stat -c %s "$part" 2>/dev/null || echo 0)"
|
||||
if [[ "$got_size" != "$want_size" ]]; then
|
||||
echo "ERROR: part $i absent or incomplete at $part (have $got_size, want $want_size)" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
for part in "$LLM_PRIMARY_DRAFT:$LLM_PRIMARY_DRAFT_SIZE" \
|
||||
"$LLM_PRIMARY_MMPROJ:$LLM_PRIMARY_MMPROJ_SIZE"; do
|
||||
local file="${part%%:*}" want="${part##*:}"
|
||||
got_size="$(stat -c %s "$file" 2>/dev/null || echo 0)"
|
||||
if [[ "$got_size" != "$want" ]]; then
|
||||
echo "ERROR: artifact absent or incomplete at $file (have $got_size, want $want)" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
[[ "$missing" -eq 0 ]] || {
|
||||
echo "frank-primary: run scripts/download-frank-primary.sh on this host" >&2
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
if ! podman image inspect "$LLM_PRIMARY_IMAGE" >/dev/null 2>&1; then
|
||||
echo "ERROR: image absent: $LLM_PRIMARY_IMAGE" >&2
|
||||
echo "frank-primary: build the fp030 image first (see the runbook)" >&2
|
||||
return 1
|
||||
}
|
||||
fi
|
||||
local want_shards have_shards
|
||||
want_shards="$(python3 - "$LLM_PRIMARY_ROOT/model.safetensors.index.json" <<'PY'
|
||||
import json, sys
|
||||
print(len(set(json.load(open(sys.argv[1]))["weight_map"].values())))
|
||||
PY
|
||||
)" || return 1
|
||||
have_shards="$(find "$LLM_PRIMARY_ROOT" -maxdepth 1 -name '*.safetensors' | wc -l)"
|
||||
if [[ -z "$want_shards" || "$have_shards" != "$want_shards" ]]; then
|
||||
echo "ERROR: payload incomplete at $LLM_PRIMARY_ROOT (have $have_shards shards, index names $want_shards)" >&2
|
||||
echo "frank-primary: run scripts/download-frank-primary-fp8.sh on this host" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
retire_lucebox_primary_quadlet
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
# A stale container with the same name makes `podman run --name` fail at
|
||||
# start; remove any leftover from a failed previous attempt.
|
||||
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
|
||||
# On a re-deploy the predecessor is THIS unit (the llama.cpp unit until
|
||||
# the 2026-09-27 cutover, the vLLM unit after it): stop it before
|
||||
# waiting on the port, or the wait deadlocks on the server it just
|
||||
# stopped serving.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
# Nothing starts until the port is released: the lucebox cutover on
|
||||
# 2026-09-26 and the frank-large cutover on 2026-09-16 both failed
|
||||
# Nothing starts until the port is released: the lucebox cutover
|
||||
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-primary
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-primary.service
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
# Cold start page-ins 175 GiB of expert pages and builds the graphs;
|
||||
# 1800s parity with the lucebox HealthStartPeriod this port has always
|
||||
# needed.
|
||||
wait_for_health frank-primary http://127.0.0.1:7770/health 1800
|
||||
# Cold start: triton/inductor compile on first boot plus reading+pinning
|
||||
# ~173 GiB of weights. Parity with every predecessor on this port.
|
||||
wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
# port cannot tell WHO answers — a stray probe holding :7770 satisfies
|
||||
# it while the unit is stopped. The deploy is only green if the unit
|
||||
# itself is active with a live MainPID behind the health answer.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-primary.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7770 answered but frank-primary.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-primary: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
|
||||
Reference in New Issue
Block a user