frank-primary: swap Dankserver to llama.cpp Qwen3.8-Flash-Next Q8_0
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
This commit is contained in:
Frank
+87
-41
@@ -1178,52 +1178,98 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the Lucebox DeepSeek V4 Quadlet
|
||||
# (the 2026-09-26 cutover that replaces frank-medium on :7770). The model
|
||||
# is a 377 GiB GGUF and the image is a local build, so the gates are:
|
||||
# image present, model file present at its declared byte count (a partial
|
||||
# download must never be served).
|
||||
# deploy_llm_primary — frank-primary is unsloth llama.cpp (the b11030-mix
|
||||
# prebuilt, ROCm gfx120X, bundled runtime) serving Qwen3.8-Flash-Next at
|
||||
# Q8_0 on the two R9700s, as a ROOT unit — the same shape as frank-large on
|
||||
# Dankclaw: a host binary, no container image, no generated unit.
|
||||
# relink_unit points /etc/systemd/system/frank-primary.service at the repo
|
||||
# file and boot persistence is that unit's [Install] section.
|
||||
#
|
||||
# The 2026-09-26 swap that replaced the Lucebox DS4.1 Quadlet on this port.
|
||||
# The payload is six Q8_0 split parts plus the shared MTP head and the
|
||||
# projector, so the gate walks every artifact at its declared byte count (a
|
||||
# partial download must never be served — and stat -c %s, never wc -c, which
|
||||
# read the whole 377 GiB lucebox file for minutes on 2026-09-26).
|
||||
LLM_PRIMARY_RUNTIME=/home/dank/GitPackages/unsloth-llama-b11030/llama-server
|
||||
LLM_PRIMARY_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
|
||||
LLM_PRIMARY_PART_SIZES=(
|
||||
10946624
|
||||
682434912
|
||||
54400261312
|
||||
49446841216
|
||||
49668930400
|
||||
34015618784
|
||||
)
|
||||
LLM_PRIMARY_DRAFT="$LLM_PRIMARY_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"
|
||||
LLM_PRIMARY_DRAFT_SIZE=2786568256
|
||||
LLM_PRIMARY_MMPROJ="$LLM_PRIMARY_ROOT/mmproj-BF16.gguf"
|
||||
LLM_PRIMARY_MMPROJ_SIZE=907542944
|
||||
|
||||
# The lucebox quadlet held this port until the 2026-09-26 cutover. Its
|
||||
# generated user unit is host state, not repo state: stop it and move the
|
||||
# installed .container into dated home trash (the same retire pattern as
|
||||
# retire_halogen_large_quadlet). The image and the 377 GiB payload stay on
|
||||
# disk as the rollback target.
|
||||
retire_lucebox_primary_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260926-llamacpp-cutover/quadlet"
|
||||
systemctl --user stop frank-primary.service 2>/dev/null || true
|
||||
if [[ -e "$quadlet_unit_dir/frank-primary.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/frank-primary.container" "$trash/frank-primary.container"
|
||||
echo "retired quadlet frank-primary.container -> $trash"
|
||||
fi
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_primary() {
|
||||
local missing=0
|
||||
local model="/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf"
|
||||
local want_size=405349859328
|
||||
if ! podman image exists localhost/lucebox-rocm:gfx1201; then
|
||||
echo "ERROR: lucebox image absent (localhost/lucebox-rocm:gfx1201)" >&2
|
||||
echo "frank-primary: podman build --build-arg LUCE_HIP_ARCHES=gfx1201 --build-arg ROCM_VERSION=7.14.1-full -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm . (see runbook)" >&2
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
local missing=0 i part want_size got_size
|
||||
if [[ ! -x "$LLM_PRIMARY_RUNTIME" ]]; then
|
||||
echo "ERROR: runtime absent at $LLM_PRIMARY_RUNTIME" >&2
|
||||
echo "frank-primary: fetch the unsloth b11030-mix gfx120X prebuilt (see the runbook)" >&2
|
||||
missing=1
|
||||
fi
|
||||
# stat -c %s, never `wc -c < FILE`. wc falls back to reading the whole
|
||||
# stream whenever its fstat shortcut does not apply, and this payload is
|
||||
# 377 GiB: on 2026-09-26 a deploy sat in wc for minutes (30 s of CPU,
|
||||
# NVMe-bound) before it could even reach the cutover.
|
||||
got_size="$(stat -c %s "$model" 2>/dev/null || echo 0)"
|
||||
if [[ "$got_size" != "$want_size" ]]; then
|
||||
echo "ERROR: model absent or incomplete at $model (have $got_size, want $want_size)" >&2
|
||||
missing=1
|
||||
fi
|
||||
[[ "$missing" -eq 0 ]] || return 1
|
||||
|
||||
# Cutover COMPLETED 2026-09-26: frank-primary took :7770 and the radiance
|
||||
# unit is retired. The host-side unit, its mask and the repo files were all
|
||||
# moved to trash by the parent session, so there is nothing left to stop or
|
||||
# mask here — masking a name that no longer exists would only resurrect it
|
||||
# in the unit search path. The port gate stays, because 7770 is a
|
||||
# host-network bind: nothing new can be created until the holder released it.
|
||||
# 7770 is a host-network bind: nothing new can be created until the old
|
||||
# holder released it (the same gate as the 2026-09-16 cutover).
|
||||
sleep 2
|
||||
wait_port_free 7770 || return 1
|
||||
|
||||
install_quadlet frank-primary dankserver/frank-primary/quadlet frank-primary
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence is the unit's [Install] WantedBy=default.target.
|
||||
systemctl --user start frank-primary.service
|
||||
# First start page-ins the dense weights and builds the graphs;
|
||||
# HealthStartPeriod in the unit is 1800s to match.
|
||||
wait_container_healthy frank-primary 1800 || {
|
||||
podman logs --tail 40 frank-primary 2>&1 || true
|
||||
for i in 1 2 3 4 5 6; do
|
||||
part="$LLM_PRIMARY_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-0000$i-of-00006.gguf"
|
||||
want_size="${LLM_PRIMARY_PART_SIZES[$((i-1))]}"
|
||||
got_size="$(stat -c %s "$part" 2>/dev/null || echo 0)"
|
||||
if [[ "$got_size" != "$want_size" ]]; then
|
||||
echo "ERROR: part $i absent or incomplete at $part (have $got_size, want $want_size)" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
for part in "$LLM_PRIMARY_DRAFT:$LLM_PRIMARY_DRAFT_SIZE" \
|
||||
"$LLM_PRIMARY_MMPROJ:$LLM_PRIMARY_MMPROJ_SIZE"; do
|
||||
local file="${part%%:*}" want="${part##*:}"
|
||||
got_size="$(stat -c %s "$file" 2>/dev/null || echo 0)"
|
||||
if [[ "$got_size" != "$want" ]]; then
|
||||
echo "ERROR: artifact absent or incomplete at $file (have $got_size, want $want)" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
[[ "$missing" -eq 0 ]] || {
|
||||
echo "frank-primary: run scripts/download-frank-primary.sh on this host" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
retire_lucebox_primary_quadlet
|
||||
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
|
||||
# the port, or the wait deadlocks on the server it just stopped serving.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
# Nothing starts until the port is released: the lucebox cutover on
|
||||
# 2026-09-26 and the frank-large cutover on 2026-09-16 both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-primary
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-primary.service
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
# Cold start page-ins 175 GiB of expert pages and builds the graphs;
|
||||
# 1800s parity with the lucebox HealthStartPeriod this port has always
|
||||
# needed.
|
||||
wait_for_health frank-primary http://127.0.0.1:7770/health 1800
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
|
||||
Reference in New Issue
Block a user