Frank
8500600d09
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
79 lines
3.7 KiB
Bash
79 lines
3.7 KiB
Bash
# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X)
|
|
# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO
|
|
# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision
|
|
# projector, and the draft-mtp + ngram-mod spec chain.
|
|
#
|
|
# These are the tunables read by scripts/serve-frank-primary.sh. The flag
|
|
# list is owned by the serve script; this file only sets fleet values.
|
|
#
|
|
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address
|
|
# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets
|
|
# away with it; do not copy the shape.
|
|
|
|
# The payload (scripts/download-frank-primary.sh): the serve script refuses
|
|
# to start without all six Q8_0 parts, the MTP head and the projector.
|
|
MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
|
|
|
|
# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same
|
|
# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm
|
|
# runtime, no host SDK gate).
|
|
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
|
|
|
# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before
|
|
# the 2026-09-26 swap, so no client, probe or models.yml entry changes
|
|
# address.
|
|
BIND_HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# One conversation, the model's full native 262144 window.
|
|
CONTEXT_SIZE=262144
|
|
PARALLEL_SLOTS=1
|
|
|
|
# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
|
|
# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
|
|
# optimum class for this pair at long contexts (the retired Dankclaw
|
|
# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
|
|
# acceptance); each extra verified position re-batches the CPU expert reads,
|
|
# so depth buys less than bandwidth here.
|
|
SPEC_DRAFT_N_MAX=2
|
|
|
|
# Device placement. -ngl 999 puts every layer's non-overridden tensors on
|
|
# the GPUs; the big overrides then move the bandwidth-huge tensors to host
|
|
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
|
# computing in RAM does not). OT_OVERRIDES is comma-separated
|
|
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
|
|
# flags:
|
|
# * exps=CPU — the routed experts. ~100 GiB at Q8, read once per token;
|
|
# the CPU backend computes them from page cache.
|
|
# * the n-gram embedding block (the 51B-param PLE table) — sparse row
|
|
# lookups, tiny PCIe traffic from CPU RAM; NEVER pin with a GPU target
|
|
# (the 23 -> 13.4 t/s regression).
|
|
# * per_layer_token_embd.weight=CPU — REQUIRED by the shared MTP head's
|
|
# borrow path: GPU-resident, it crashes the draft load with
|
|
# "invalid vector subscript" (llama.cpp PR #28243).
|
|
# The exact tensor pattern for the n-gram block was discovered at load time
|
|
# on 2026-09-26 (see the runbook); this list is the deployed set.
|
|
NGL=999
|
|
OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU
|
|
TENSOR_SPLIT=1,1
|
|
|
|
# The drafter lives on the first R9700 (the card the load log shows holding
|
|
# the shared embedding borrow path).
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
|
|
# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
|
|
# class on this fleet's ROCm stack (a bigger logical batch is free; compute
|
|
# buffers reserve at max graph size). ubatch 512 balances prefill
|
|
# throughput against the asymmetric compute-buffer cost on the card that
|
|
# holds the output layers and the drafter.
|
|
BATCH_SIZE=8192
|
|
UBATCH_SIZE=512
|
|
|
|
# Prompt-cache budget: 16 GiB. The KV itself is small (the hybrid
|
|
# DeltaNet/QSA pattern keeps a 262144 window a fraction of the full-ctx
|
|
# x all-layers number), so the prompt cache is the RAM-side spend that
|
|
# actually moves prefill on repeat traffic. Host RAM budget: ~175 GiB of
|
|
# expert pages + this cache + the server's own arenas inside 246 GiB.
|
|
CACHE_RAM=16384
|