Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest
dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the
.hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2)
on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law).
Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images)
and the censored base (rollback = revert this commit; payload, fork
build and build-rocmfpx.sh pin stay on disk).
- serve script: podman run under the root unit, exec podman wait so
Restart=always is real; gates image digest, checkpoint byte-count,
flat regular-file tokenizer, GPU nodes; engine stdout to log.
- deploy handler: gates the artifacts, sweeps the legacy bare
llama-server + old container name by NAME (never pkill llama-server:
frank-small), retires the 2026-09 quadlet, port cutover 7770.
- fetch script: -C - resume, -f, byte count, sha256, --verify-only.
- seam rewritten (17 tests): halogen is now the deployed runtime.
The repo's highest-quality rung is ftype 115: 311 tensors of
GGML_TYPE_Q8_0_ROCMFPX, a type only the ROCmFPX fork reads, so the runtime
moves from gufo to scripts/build-rocmfpx.sh's pinned fork build (36e9acd) —
the first build on this host measured to load this file without the
2026-08-25 set_rows segfault. One session at the native 262,144 window, f16
KV (64 KiB/token, 16 GiB), the repo's Q4_0 MTP head drafting 4 tokens, plus
the bundled BF16 projector so the endpoint has vision again.
The serve script now honors caller environment overrides over the env file
(a plain source let the file win, which made single-run overrides no-ops) and
defaults apply only when a value is unset, so an explicit LLAMA_BIN='' is the
error it reads as.
Engine: digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 (vLLM 0.30,
TheRock ROCm 10, Triton GDN kernels); the serve script gates the image by
REPO DIGEST, the payload (config + index + exact shard count), the spec
draft (BOTH weight layouts: single model.safetensors for dflash, sharded
index for the Qwen3.5-9B MTP head), and whole /dev/dri, sweeps the bench
identity vllm-mimo, then exec podman wait so a container death trips
Restart=always. Shape: 4 x 262144 at util 0.45 (KV grows only on the 8
full-attention layers), MTP 2 draft tokens.
Measured 2026-10-01 window (frank-large stopped): base 13.1 t/s decode
(~94% of the LPDDR5x BF16 roof), DFlash-8 17.4, MTP-2 19.8 with 71%/44%
per-position acceptance - the cross-model head works because vLLM reads the
draft from a separate path while MiMo ships zero mtp.* tensors. Spec arms
do not preserve greedy bf16 text (batched-verify reduction order).
The seam was rewritten around the new laws (18 tests): unit self-healing,
digest gating, engine wiring, the measured spec table, the retired gufo
gates, the deployer MainPID check. frank_small port test follows the
PORT -> FRANK_LARGE_PUBLISH rename. Rollback: tag frank-large-gufo-q4kxl-20261001.
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.
Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.
Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.
Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.
The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.
Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
Replace the CIRU v4.4.1 Orca runtime with the unsloth llama.cpp prebuilt
(b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB, three HF split parts
concatenated) on the Strix Halo iGPU. One conversation, 200,000 ctx,
PARALLEL_SLOTS=1, MTP depth 2 via the explicit -md shared-Q8_0 head
(sidecar auto-discovery does not search the mtp/ subfolder), text-only.
Fleet contract unchanged: port 7770, alias frank-large.
- serve-frank-large.sh: unsloth binary, -m/-md, --spec-type draft-mtp
--spec-draft-n-max 2, --jinja, preserve_thinking; gates on binary,
GGUF, MTP head; conditional --mmproj on ENABLE_VISION=1
- download-frank-large-unsloth.sh: resumable fetch of 3 parts + MTP head
+ mmproj, each verified against its LFS sha256 fetched live from the
Hub tree API; concatenates parts, records/verifies SHA256SUMS
- deploy-remote.sh: LLM_LARGE_MODEL/LLM_LARGE_DRAFT paths, pkill pattern
on the model path (not the binary name), wait_port_free gate
- tests: seams for 200k/1-slot, explicit -md, text-only, handler gates,
sweep safety
Operator call 2026-09-19: switch the iGPU endpoint from
halogen-flash-server to jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the CIRU
v4.4.1 runtime with the QSA window hotfix (host build against the private
ROCm 10 SDK), one stream of the full 262144 trained window. The service
leaves the quadlet shape for a root unit (the same shape as frank-small and
frank-medium: a private-SDK host binary has no image to install and no
generated unit to dodge); the halogen quadlet is archived, the deploy
handler retires it on the host, gates on 7770, and its stray-run sweep
matches the model path so it cannot take frank-small down.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Frank
