feat(llm): swap frank-large to halogen-flash-server, retire frank-npu

frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
This commit is contained in:
Frank
2026-09-11 14:48:28 -07:00
parent 184df9aa0d
commit 301914b1c0
21 changed files with 929 additions and 279 deletions
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# Serve the FastFlowLM (FLM) sight+sound endpoint — frank-npu.
#
# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
# single-lane device: generations serialize in FIFO order, so this is a
# background/triage endpoint, not an interactive multi-user chat. The NPU
# draws from the unified RAM pool shared with frank-large (the iGPU
# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
# for the memory budget and the model-selection tradeoff.
#
# NOTE: FLM requires unlimited memlock. The unit sets LimitMEMLOCK=infinity.
# NOTE: an unknown model tag does NOT 404 — /v1/chat/completions answers HTTP 200 with a
# flat {"error":"Failed to load <tag> model!"} body and no choices; /v1/completions returns a
# nested OpenAI-shaped server_error (code 500) also under HTTP 200. Client model IDs must be the exact FLM registry
# tag (e.g. gemma4-it:e4b), never a service alias. Custom tags only resolve
# when FLM_CONFIG_PATH points at the user registry that carries them and their
# kernels are symlinked into /home/dank/.config/flm/xclbins/<PackName>.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_NPU_ENV_FILE:-$SCRIPT_DIR/frank-npu.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) -------------------
: "${FLM_BIN:=/usr/bin/flm}"
: "${FLM_MODEL:=gemma4-it:e4b}"
: "${FLM_HOST:=0.0.0.0}"
: "${FLM_PORT:=7771}"
: "${FLM_PMODE:=turbo}"
: "${FLM_ASR:=0}"
: "${FLM_CTX_LEN:=131072}"
# The user registry is a superset of the system registry (it carries the
# custom tags too), and its model_path resolves relative to the registry file,
# so exports keep a tag swap in the env file from needing script edits.
export FLM_CONFIG_PATH="${FLM_CONFIG_PATH:-/home/dank/.config/flm/model_list.json}"
export FLM_XCLBIN_PATH="${FLM_XCLBIN_PATH:-/home/dank/.config/flm/xclbins}"
if [[ ! -x "$FLM_BIN" ]]; then
echo "missing flm binary: $FLM_BIN" >&2
exit 1
fi
# --- optional flags ----------------------------------------------------------
OPT_ARGS=()
if [[ "${FLM_ASR}" != "0" ]]; then
OPT_ARGS+=(--asr "${FLM_ASR}")
fi
if [[ "${FLM_CTX_LEN}" != "-1" ]]; then
OPT_ARGS+=(--ctx-len "${FLM_CTX_LEN}")
fi
exec "$FLM_BIN" serve "$FLM_MODEL" \
--host "$FLM_HOST" \
-p "$FLM_PORT" \
--pmode "$FLM_PMODE" \
"${OPT_ARGS[@]}"