frank-large: cut over to halogen container (qwen3.8-27b .hgn + DFlash2)

Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the .hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2) on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law). Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images) and the censored base (rollback = revert this commit; payload, fork build and build-rocmfpx.sh pin stay on disk). - serve script: podman run under the root unit, exec podman wait so Restart=always is real; gates image digest, checkpoint byte-count, flat regular-file tokenizer, GPU nodes; engine stdout to log. - deploy handler: gates the artifacts, sweeps the legacy bare llama-server + old container name by NAME (never pkill llama-server: frank-small), retires the 2026-09 quadlet, port cutover 7770. - fetch script: -C - resume, -f, byte count, sha256, --verify-only. - seam rewritten (17 tests): halogen is now the deployed runtime.
2026-10-05 16:29:11 -07:00
1 parent aa9908257d
commit fece3ffdd0
@@ -1,186 +1,133 @@
 #!/usr/bin/env bash
-# Serve kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF as
-# frank-large on Dankclaw's Strix Halo iGPU (gfx1151) with the ROCmFPX fork of
-# llama.cpp.
+# serve-frank-large.sh — frank-large endpoint: halogen (closed-source
+# C++/HIP engine, image ghcr.io/peonist-ai/halogen, digest-pinned in
+# frank-large.env) serving peonist-ai/halogen-qwen3.8-27b on Dankclaw's
+# Strix Halo iGPU, host :7770. 2026-10-05 cutover from the ROCmFPX
+# llama.cpp shape — revert the swap commit's trio to roll back (payload
+# and fork build stay on disk).
 #
-# 2026-10-03 (Frank): swapped from the ALIGNED Q8_0_ROCMFPX_AGENT to this
-# repo's Q6_0_ROCMFPX_AGENT — abliterated weights, one quant rung cheaper,
-# everything else (window, KV, MTP depth, sampler) unchanged.
+# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
+# pattern): the image entrypoint `halogen-docker` in default "all" mode
+# starts engine + OpenAI frontend in one container. The frontend listens on
+# 8731 inside (HALOGEN_API_PORT default) and is published as host 7770; the
+# engine's own token port (8730, no auth) stays unpublished on the
+# container's loopback. The image ships NO default checkpoint:
+# HALOGEN_CHECKPOINT selects the file inside the /models mount.
 #
-# Why this runtime and not another: the payload is ftype 114 — 339 tensors of
-# GGML_TYPE_Q6_0_ROCMFPX (102) and 159 of GGML_TYPE_Q8_0_ROCMFPX (103), tensor
-# types that exist only in the ROCmFPX fork. Stock llama.cpp, SGLang and gufo
-# all refuse the file, and gufo additionally gates every "Qwen3.8" artifact on
-# a template-hash allowlist. The qualified build is scripts/build-rocmfpx.sh's
-# pin: it defines both types (ggml.h 102/103), it carries the device paths for
-# type 102 (convert/cpy/getrows/mmq/fattn in ggml-cuda), it survives warmup —
-# the 2026-08-25 builds of the older fork line segfaulted in
-# ggml_compute_forward_set_rows on this family — and it drafts with the
-# model's own MTP head.
+# Gates mirror the deployer EXACTLY (the 2026-10-01 crash-loop lesson —
+# serve gate and deploy gate disagreeing = green deploy, red unit):
+# image digest present; checkpoint present at the exact byte count;
+# tokenizer present, readable, and a FLAT REGULAR FILE — the engine repo
+# warns that a HF cache snapshot's symlinks into blobs/ dangle inside a
+# container; GPU device nodes present.
 #
-# Why this file: Q6_0_ROCMFPX_AGENT (23.15 GiB, ftype 114) is the card's
-# protected-head rung — "Take Q6 AGENT (114) if you want protected heads": the
-# AGENT routing keeps the embeddings, the output head and every fourth
-# attention projection at the Q8 rung (159 tensors), which is the variant the
-# card names for MTP serving, and the refusal study was scored on this exact
-# file (23/24 comply where the aligned Q8 AGENT refuses 23/24, quality score
-# identical at 6/8). Drafting uses the repo's Q4_0 MTP head: the head's cost is
-# its projection over the 248K vocabulary, so the 4-bit head is faster end to
-# end at equal-or-better acceptance than its Q8_0 sibling.
-#
-# The client contract lives here, not in frank-large.env:
-#   * --jinja — without it llama.cpp silently IGNORES the harness's
-#     chat_template_kwargs (enable_thinking / preserve_thinking), so the
-#     endpoint would look like it had no thinking control at all. The GGUF
-#     carries the official Qwen template, so no --chat-template-file.
-#   * --reasoning-format deepseek — thinking is emitted as reasoning_content,
-#     the field the harness reads for this endpoint.
-#   * the template accepts reasoning_effort xhigh|medium|low only, which is
-#     why models.yml folds the harness's levels onto that set.
-#   * --alias frank-large — that is the model id the harness resolves.
-#   * --ctx-checkpoints / --cache-ram — prompt-cache geometry for the 16
-#     sliding-window-style layers; harmless where the hybrid state forces a
-#     full re-process, and required for the checkpoint feature to work at all.
+# Why containers instead of the bare binary (the old frank-large shape):
+# the payload is .hgn and only halogen reads it; halogen ships as a
+# container; the root unit keeps LimitMEMLOCK=infinity for parity and to
+# outlive the login session, exactly like frank-primary's.
 set -euo pipefail

-# Fleet values live in frank-large.env. Anything already in the environment
-# wins over the file, so an operator can override one value on a single run:
-# the caller's values are captured, the file is sourced, then they are put
-# back. (A plain `source` would let the file clobber the environment.)
-TUNABLES=(LLAMA_BIN MODEL_FILE DRAFT_FILE MMPROJ_FILE ALIAS BIND_ADDR
-          LISTEN_PORT CTX_SIZE PARALLEL BATCH_SIZE UBATCH_SIZE NGL FIT
-          FLASH_ATTN CACHE_TYPE_K CACHE_TYPE_V SPEC_TYPE SPEC_DRAFT_NGL
-          SPEC_DRAFT_DEVICE SPEC_DRAFT_N_MAX SPEC_DRAFT_N_MIN SPEC_DRAFT_P_MIN
-          TEMPERATURE TOP_P TOP_K MIN_P PRESENCE_PENALTY NUM_THREADS
-          THREAD_BATCH ROCM_PATH HIP_VISIBLE HSA_GFX_VERSION)
-_OVERRIDES=()
-for _var in "${TUNABLES[@]}"; do
-    [[ -n "${!_var+set}" ]] && _OVERRIDES+=("$_var=${!_var}")
-done
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$(dirname "$0")/frank-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
+if [ ! -f "$ENV_FILE" ]; then
+    echo "frank-large: env file not found: $ENV_FILE" >&2
+    exit 1
 fi
-for _kv in "${_OVERRIDES[@]+${_OVERRIDES[@]}}"; do
-    export "${_kv?}"
-done
-unset _var _kv
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a

-: "${LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
-: "${MODEL_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
-: "${DRAFT_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mtp-Qwen3.8-27B-Q4_0.gguf}"
-: "${MMPROJ_FILE=/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/mmproj-Qwen3.8-27B-BF16.gguf}"
-: "${ALIAS=frank-large}"
-: "${BIND_ADDR=0.0.0.0}"
-: "${LISTEN_PORT=7770}"
-: "${CTX_SIZE=262144}"
-: "${PARALLEL=1}"
-: "${BATCH_SIZE=2048}"
-: "${UBATCH_SIZE=512}"
-: "${NGL=999}"
-: "${FIT=off}"
-: "${FLASH_ATTN=on}"
-: "${CACHE_TYPE_K=f16}"
-: "${CACHE_TYPE_V=f16}"
-: "${SPEC_TYPE=draft-mtp}"
-: "${SPEC_DRAFT_NGL=99}"
-: "${SPEC_DRAFT_DEVICE=ROCm0}"
-: "${SPEC_DRAFT_N_MAX=4}"
-: "${SPEC_DRAFT_N_MIN=0}"
-: "${SPEC_DRAFT_P_MIN=0.0}"
-: "${TEMPERATURE=1.0}"
-: "${TOP_P=0.95}"
-: "${TOP_K=20}"
-: "${MIN_P=0.0}"
-: "${PRESENCE_PENALTY=0.0}"
-: "${NUM_THREADS=16}"
-: "${THREAD_BATCH=16}"
-: "${ROCM_PATH=/opt/rocm}"
-: "${HIP_VISIBLE=0}"
-: "${HSA_GFX_VERSION=11.5.1}"
+: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
+: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
+: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
+: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
+: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
+: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
+: "${FRANK_LARGE_TOKENIZER:?FRANK_LARGE_TOKENIZER must name the tokenizer file}"
+: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
+: "${HALOGEN_SLOT_CTX:?HALOGEN_SLOT_CTX must be set}"
+: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set (1 keeps the drafter on)}"
+: "${HALOGEN_DRAFTER:?HALOGEN_DRAFTER must be set (2 = DFlash2)}"

-[[ -n "$LLAMA_BIN" ]] || { echo "frank-large: LLAMA_BIN is empty" >&2; exit 1; }
-[[ -x "$LLAMA_BIN" ]] || {
-    echo "frank-large: llama-server '$LLAMA_BIN' is not executable;" >&2
-    echo "           build it with scripts/build-rocmfpx.sh" >&2
+# Capture podman's output BEFORE grepping: under `set -o pipefail` the
+# `podman images | grep -q` pattern races (grep exits on first match,
+# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
+# 2026-09-30 deploy refusal).
+DIGESTS="$(podman images --format '{{.RepoDigests}}')"
+if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
+    echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
+    echo "  pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
    exit 1
-}
-# Refuse anything but the real payload. A path that is not a readable file
-# makes llama-server hang rather than fail clean.
-[[ -r "$MODEL_FILE" ]] || {
-    echo "frank-large: model file '$MODEL_FILE' is not readable;" >&2
-    echo "           fetch it with scripts/download-frank-large-kingjones-uncensored.sh" >&2
-    exit 1
-}
-# The MTP head is the performance feature: without it these weights decode at
-# memory-bandwidth speed (measured on the aligned sibling: 17.0 t/s with the
-# head against 7.6 without) and the endpoint loses the ~2.2x the head buys.
-[[ -r "$DRAFT_FILE" ]] || {
-    echo "frank-large: MTP draft head '$DRAFT_FILE' is not readable;" >&2
-    echo "           fetch it with scripts/download-frank-large-kingjones-uncensored.sh" >&2
-    exit 1
-}
-# The GGUF carries no vision tower; the projector file is the whole tower, so
-# a missing file means a vision request would 500 rather than degrade.
-# MMPROJ_FILE="" is the documented opt-out for a text-only endpoint.
-MM_ARGS=()
-if [[ -n "${MMPROJ_FILE:-}" ]]; then
-    [[ -r "$MMPROJ_FILE" ]] || {
-        echo "frank-large: mmproj '$MMPROJ_FILE' is not readable; set MMPROJ_FILE='' for text-only" >&2
-        exit 1
-    }
-    MM_ARGS=(--mmproj "$MMPROJ_FILE")
 fi

-# HIP/ROCm host environment. HIP reads bare env vars (TEMP, HOST,
-# GPU_MAX_HEAP_SIZE), which is why the tunables above are named TEMPERATURE
-# and BIND_ADDR. HSA_OVERRIDE_GFX_VERSION is what lets a runtime built for a
-# neighbouring gfx target run on this iGPU; the qualified pin targets gfx1151
-# natively and the override is a no-op for it.
-export ROCM_PATH
-export HIP_VISIBLE_DEVICES="$HIP_VISIBLE"
-export HSA_OVERRIDE_GFX_VERSION="$HSA_GFX_VERSION"
-export LD_LIBRARY_PATH="$(dirname "$LLAMA_BIN"):${ROCM_PATH}/lib:${ROCM_PATH}/lib/rocm_sysdeps/lib:${ROCM_PATH}/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+# The checkpoint gate is size-exact: a truncated curl exits 0 on some
+# failures and a partial .hgn loads far enough to look alive, so the byte
+# count from the model card is the contract (the fleet fetch law).
+CKPT="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT"
+if [ ! -f "$CKPT" ]; then
+    echo "frank-large: checkpoint $CKPT missing;" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    exit 1
+fi
+CKPT_SIZE="$(stat -c %s "$CKPT")"
+if [ "$CKPT_SIZE" != "$FRANK_LARGE_CHECKPOINT_BYTES" ]; then
+    echo "frank-large: checkpoint is $CKPT_SIZE bytes, expected $FRANK_LARGE_CHECKPOINT_BYTES (partial download?)" >&2
+    exit 1
+fi

-exec /usr/bin/env "$LLAMA_BIN" \
-    --model "$MODEL_FILE" \
-    "${MM_ARGS[@]}" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    --jinja \
-    --device ROCm0 \
-    --n-gpu-layers "$NGL" \
-    --parallel "$PARALLEL" \
-    --ctx-size "$CTX_SIZE" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --threads "$NUM_THREADS" \
-    --threads-batch "$THREAD_BATCH" \
-    --fit "$FIT" \
-    --flash-attn "$FLASH_ATTN" \
-    --direct-io \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --ctx-checkpoints 16 \
-    --cache-ram 8192 \
-    --cache-idle-slots \
-    --temp "$TEMPERATURE" \
-    --top-p "$TOP_P" \
-    --top-k "$TOP_K" \
-    --min-p "$MIN_P" \
-    --presence-penalty "$PRESENCE_PENALTY" \
-    --reasoning on \
-    --reasoning-format deepseek \
-    --reasoning-budget -1 \
-    --spec-type "$SPEC_TYPE" \
-    --model-draft "$DRAFT_FILE" \
-    --spec-draft-ngl "$SPEC_DRAFT_NGL" \
-    --spec-draft-device "$SPEC_DRAFT_DEVICE" \
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
-    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
-    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
-    --metrics
+# The tokenizer must be a regular file in a FLAT directory: pointing the
+# mount at a HuggingFace cache snapshot hands the container symlinks into
+# a sibling blobs/ that dangle inside the mount.
+TOK="$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_TOKENIZER"
+if [ ! -f "$TOK" ] || [ -L "$TOK" ]; then
+    echo "frank-large: tokenizer $TOK missing or a symlink (HF cache snapshot?);" >&2
+    echo "  fetch it: bash scripts/download-frank-large-halogen.sh" >&2
+    exit 1
+fi
+
+# GPU access is mandatory but the node paths are variables so the seam can
+# drive the gate on a box without an AMD GPU.
+DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
+DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
+if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
+    echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
+    exit 1
+fi
+for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done
+
+LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
+mkdir -p "$(dirname "$LOG")"
+# for the udev-granted supplementary group, seccomp unconfined for the ROCm
+# ioctl surface, and unbounded memlock/nofile (the engine mmaps 33.4 GiB of
+# weights and wants a big fd budget). No comment lines inside the backslash
+# chain — a mid-chain # terminates the continuation (the 2026-10-05
+# exit-127 crash loop on the sibling serve script).
+podman run -d --name "$FRANK_LARGE_CONTAINER" \
+  --publish "$FRANK_LARGE_PUBLISH" \
+  --ipc=host \
+  --ulimit memlock=-1:-1 --ulimit nofile=1048576:1048576 \
+  --device "$DEV_KFD" --device "$DEV_DRI" \
+  --group-add keep-groups \
+  --security-opt seccomp=unconfined \
+  -e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
+  -e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
+  -e HALOGEN_TOKENIZER="/tokenizer/$FRANK_LARGE_TOKENIZER" \
+  -e HALOGEN_SLOT_CTX="$HALOGEN_SLOT_CTX" \
+  -e HALOGEN_KV_SLOTS="${HALOGEN_KV_SLOTS:-1}" \
+  -e HALOGEN_DRAFTER="$HALOGEN_DRAFTER" \
+  -e HALOGEN_MAX_TOKENS_CAP="${HALOGEN_MAX_TOKENS_CAP:-65536}" \
+  -e HALOGEN_QUEUE_TIMEOUT="${HALOGEN_QUEUE_TIMEOUT:-7200}" \
+  ${HALOGEN_CACHE_MB:+-e HALOGEN_CACHE_MB="$HALOGEN_CACHE_MB"} \
+  -v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
+  -v "$FRANK_LARGE_MODEL_DIR:/tokenizer:ro" \
+  "$FRANK_LARGE_IMAGE" \
+  > "$LOG" 2>&1
+
+# This script's lifetime IS the container's: podman wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=always is real. systemd stop goes through the unit's ExecStop
+# (podman wait alone cannot stop the child).
+exec podman wait "$FRANK_LARGE_CONTAINER"