Frank
d4ce76b68b
Every comment that invoked a 'law' (50 word-boundary sites across 28 tracked files) now states what is going on and why, in terms that transfer to any project: gate twins drift until one side greenlights what the other refuses; a tag re-points while a digest pins; a restrictive engine behavior belongs to the runtime that imposed it and must be re-tested, not inherited; a speculative head is tensors in the file, not a flag; snapshot links dangle where the container resolves them; exit 0 is a clean stop to systemd, so a self-exiting daemon on on-failure goes dark reporting success. Event narratives and dates are gone from these comments; the mechanics carry the argument. Behavior-identical: bash -n, py_compile, yaml parse, full bats suite 408/408 green. One symbol rename rides along (stack script local scale_law -> scale_contract, with its log text).
206 lines
11 KiB
Bash
Executable File
206 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
|
|
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
|
# NATIVE block-FP8 precision from a rad-convert container
|
|
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
|
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
|
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
|
|
# pre-cutover state).
|
|
#
|
|
# The run line below IS the engine argv: the image's entrypoint is the
|
|
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
|
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
|
|
# log states every budget before anything is allocated (weights vs KV
|
|
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
|
# the P2P answer per pair) — read it after every resize.
|
|
# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
|
|
# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
|
|
# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
|
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
|
|
#
|
|
# The root unit (scripts/frank-archon.service, User=dank,
|
|
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
|
|
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
|
|
# pages, which is ALSO why this script refuses to start when ttm's
|
|
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
|
|
# The artifact gates are the shared library: the serve script and the
|
|
# deployer gate the same payload, and two hand-written copies of the same
|
|
# checks drift apart until one side passes what the other refuses. One
|
|
# library, one refusal text, one rule.
|
|
export GATE_PREFIX=frank-archon
|
|
# shellcheck disable=SC1091
|
|
source "$SCRIPT_DIR/lib/llm-gates.sh"
|
|
ENV_FILE="${FRANK_ARCHON_ENV_FILE:-$SCRIPT_DIR/frank-archon.env}"
|
|
|
|
if [ ! -f "$ENV_FILE" ]; then
|
|
echo "ERROR: frank-archon env file absent: $ENV_FILE" >&2
|
|
exit 1
|
|
fi
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
set +a
|
|
|
|
: "${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
|
: "${FRANK_ARCHON_MODEL_DIR:?FRANK_ARCHON_MODEL_DIR must name the .rad dir}"
|
|
: "${FRANK_ARCHON_MODEL_FILE:?FRANK_ARCHON_MODEL_FILE must name the .rad file}"
|
|
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the /data state dir}"
|
|
: "${FRANK_ARCHON_CONTAINER:?FRANK_ARCHON_CONTAINER must name the container}"
|
|
: "${FRANK_ARCHON_SERVED_NAME:?FRANK_ARCHON_SERVED_NAME must name the alias}"
|
|
: "${FRANK_ARCHON_PORT:?FRANK_ARCHON_PORT must be the port}"
|
|
: "${FRANK_ARCHON_TEMPLATE:?FRANK_ARCHON_TEMPLATE must name the chat template}"
|
|
: "${FRANK_ARCHON_TP:?FRANK_ARCHON_TP must be set}"
|
|
: "${FRANK_ARCHON_TP_WIRE:?FRANK_ARCHON_TP_WIRE must be set}"
|
|
: "${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
|
|
: "${FRANK_ARCHON_MAX_MODEL_LEN:?FRANK_ARCHON_MAX_MODEL_LEN must be set}"
|
|
: "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
|
|
: "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
|
|
: "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
|
|
: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
|
|
: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
|
|
: "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
|
|
: "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
|
|
: "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
|
|
: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
|
|
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
|
|
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
|
|
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
|
|
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
|
|
|
|
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
|
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
|
# newer latest does not satisfy it. One command, no pipefail/grep race (the
|
|
# 2026-09-30 lesson). Runs as dank: the unit's process uses dank's ROOTLESS
|
|
# podman storage (root unit, User=dank — the fleet's frank-archon shape).
|
|
require_image "$FRANK_ARCHON_IMAGE" \
|
|
"podman pull docker://stilldeadcode/radiance:latest (as dank)" || exit 1
|
|
|
|
# The payload gate: the .rad container file, sized. A truncated convert
|
|
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
|
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
|
# (the convert script built the file; its rad-info output is the record).
|
|
require_min_mib "$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE" \
|
|
container "$FRANK_ARCHON_RAD_MIN_MIB" \
|
|
"build it: scripts/convert-swift15-w4dn8.sh (rad-info is the record)" || exit 1
|
|
|
|
require_file "$FRANK_ARCHON_TEMPLATE" template \
|
|
"the chat template is vendored: scripts/qwen38-froggeric-v22.5.jinja" || exit 1
|
|
|
|
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
|
|
# The engine's create is NOT recursive: with only the state dir mounted, its
|
|
# "cannot create /data/kvcache/frank-archon: No such file or directory"
|
|
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
|
|
# the full path on the host so the tier opens on first boot.
|
|
|
|
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
|
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
|
|
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
|
|
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
|
|
# the cache must ride a host mount or it dies with the --rm container
|
|
# that wrote it. The dir is pre-created for the same reason as kvcache
|
|
# (the engine's create is not recursive).
|
|
|
|
# GPU access is mandatory; the node paths are knobs so the seam can drive
|
|
# this gate on a box without AMD GPUs (production defaults).
|
|
DEV_KFD="${FRANK_ARCHON_DEV_KFD:-/dev/kfd}"
|
|
DEV_DRI="${FRANK_ARCHON_DEV_DRI:-/dev/dri}"
|
|
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
|
echo "ERROR: GPU device nodes absent ($DEV_KFD, $DEV_DRI)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
|
|
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
|
# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
|
|
# the load dies mid-pin with an allocator error instead of this named
|
|
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
|
|
# fallback + one reboot only if the file is read-only), see the runbook.
|
|
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
|
|
ttm_cap_mib() {
|
|
# min over the knobs THIS kernel exposes — the current kernel has
|
|
# pages_limit only (no pages_max); a knob that is absent is skipped,
|
|
# not a zero (a zero would refuse a healthy cap).
|
|
local v cur="" dir
|
|
dir="${FRANK_ARCHON_TTM_SYSFS:-/sys/module/ttm/parameters}"
|
|
for p in "$dir/pages_limit" "$dir/pages_max"; do
|
|
[ -r "$p" ] || continue
|
|
v="$(cat "$p" 2>/dev/null)" || continue
|
|
[ -n "$v" ] || continue
|
|
v=$(( v * 4 / 1024 ))
|
|
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
|
done
|
|
echo "${cur:-0}"
|
|
}
|
|
TTM_CAP="$(ttm_cap_mib)"
|
|
if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
|
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
|
echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# A crashed/killed unit can leave the container name registered (systemd
|
|
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
|
# identities (the FreeToken-era frank-archon-qwen, the retired R9V
|
|
# container and the ad hoc experiment containers all hold this GPU); all
|
|
# idempotent.
|
|
for CTR in $FRANK_ARCHON_LEGACY_CONTAINERS "$FRANK_ARCHON_CONTAINER"; do
|
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
|
done
|
|
|
|
LOG="${FRANK_ARCHON_LOG:-/home/dank/logs/frank_archon.log}"
|
|
mkdir -p "$(dirname "$LOG")"
|
|
|
|
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
|
# --network host (the server binds the host's addresses directly, no NAT),
|
|
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
|
|
# (Docker/podman's default profile refuses io_uring, which the n-gram
|
|
# table's reads go through, and the NUMA binding calls the ROCm runtime
|
|
# makes for host memory), memlock unbounded (the pinned ledger — legal
|
|
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
|
|
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
|
|
# The template rides a single-file mount; the .rad dir is ro; /data holds
|
|
# the prefix-cache disk tier.
|
|
podman run -d --name "$FRANK_ARCHON_CONTAINER" \
|
|
--network host \
|
|
--init \
|
|
--security-opt seccomp=unconfined \
|
|
--ulimit memlock=-1:-1 \
|
|
--device /dev/kfd --device /dev/dri \
|
|
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
|
|
-v "$FRANK_ARCHON_STATE_DIR":/data \
|
|
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
|
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
|
|
"$FRANK_ARCHON_IMAGE" \
|
|
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
|
|
--host :: --port "$FRANK_ARCHON_PORT" \
|
|
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
|
|
--override-chat-template /templates/template.jinja \
|
|
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
|
|
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
|
|
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
|
|
--placement "$FRANK_ARCHON_PLACEMENT" \
|
|
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
|
|
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
|
|
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
|
|
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
|
|
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
|
|
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
|
|
--prefix-cache-dir /data/kvcache/frank-archon \
|
|
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
|
|
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
|
|
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
|
|
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
|
|
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
|
|
> "$LOG" 2>&1
|
|
|
|
# From here on, this script's lifetime IS the container's: wait returns the
|
|
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
|
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
|
# podman wait alone cannot stop the container child.)
|
|
exec podman wait "$FRANK_ARCHON_CONTAINER"
|