Frank
d57174c8d3
Three engine knobs that were impossible to set without editing the run line:
FRANK_PRIMARY_SPEC (--speculative-config), FRANK_PRIMARY_COMPCONFIG
(--compilation-config) and FRANK_PRIMARY_GEN_CONFIG
(--override-generation-config). Each holds the whole JSON value; empty or
unset means the flag is absent, so the shipped shape E argv is unchanged.
The seam re-pins the law that justified the omission: "speculative decode is
ABSENT BY MEASUREMENT" compared the wrong pair — c1248d4 retired MTP by
weighing fp8+MTP4 (37.4-40.6 t/s, 54% acceptance) against the bf16 no-spec
boot (36.4-39.4), bf16 then proved unbootable, and fp8 came back WITHOUT MTP.
fp8+spec was never measured against fp8+no-spec on one shape, the no-spec
incumbent runs ~30 t/s under the fleet's 35 t/s target, and every vendor
flash-next recipe ships MTP3.
Also in the seam: a dead assertion is deleted (a failing `! grep` mid-test is
exempt from bats' errexit, so the pre-precise-memory `! grep
--expert-precise-memory` never ran), and the knob test asserts by RENDERING the
flag from the sourced env instead of grepping for an assignment.
169 lines
8.9 KiB
Bash
Executable File
169 lines
8.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
|
|
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
|
|
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
|
|
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
|
|
#
|
|
# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
|
|
# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
|
|
# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
|
|
# a 104-key on-host setup state; this stack is fully expressed by flags, and
|
|
# every flag here was qualified by boot probe on 2026-09-29 — see
|
|
# frank-primary.env for the qualification record).
|
|
#
|
|
# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
|
|
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
|
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
|
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
|
# CANNOT boot this model
|
|
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
|
|
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
|
|
# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
|
|
# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
|
|
# retirement of MTP compared the wrong pair — see frank-primary.env.
|
|
# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
|
|
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
|
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
|
|
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
|
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
|
|
# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to
|
|
# git tag frank-primary-fp8-mtp-20261001.
|
|
#
|
|
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
|
|
# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
|
|
# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
|
|
# container's ephemeral layer, and `podman rm -f` deletes them — so every
|
|
# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
|
|
# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
|
|
# later boots load the compiled artifact instead, and hides nothing. PROVEN
|
|
# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
|
|
# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
|
|
# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
|
|
# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
|
|
#
|
|
# The root unit (scripts/frank-primary.service, User=dank,
|
|
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
|
# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
|
|
# also works, but the unit is the fleet-parity boot-persistent shape.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
|
|
|
|
if [ ! -f "$ENV_FILE" ]; then
|
|
echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
|
|
exit 1
|
|
fi
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
set +a
|
|
|
|
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
|
|
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
|
|
: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
|
|
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
|
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
|
: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
|
|
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
|
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
|
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
|
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
|
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
|
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
|
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
|
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
|
# classic `podman images | grep -q` pattern races — grep exits on its first
|
|
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
|
# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
|
|
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
|
if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
|
|
echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
|
|
echo " fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
|
|
echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
|
|
exit 1
|
|
fi
|
|
SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
|
if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
|
|
echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
|
|
echo " found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
|
|
echo " re-fetch: scripts/download-frank-primary.sh" >&2
|
|
exit 1
|
|
fi
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
|
|
echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
|
echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# A crashed/killed unit can leave the container name registered (systemd
|
|
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
|
# identities (the retired R9V container and the ad hoc experiment container
|
|
# both hold these GPUs); all idempotent.
|
|
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
|
done
|
|
|
|
LOG=/home/dank/logs/frank_primary.log
|
|
mkdir -p "$(dirname "$LOG")"
|
|
|
|
# Persistent compile cache (see the header). Create it before the mount so even
|
|
# the first, cold boot has a target; unset -> no mount, exactly the old
|
|
# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
|
|
# /root/.cache/comgr (HIP kernel JIT).
|
|
CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
|
|
if [ -n "$CACHE_DIR" ]; then
|
|
mkdir -p "$CACHE_DIR"
|
|
fi
|
|
|
|
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
|
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
|
# available" while device_count() says 2). --ipc=host for the engine's shared
|
|
# memory; memlock unbounded for the pinned host slabs.
|
|
# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
|
|
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|
--publish "$FRANK_PRIMARY_PUBLISH" \
|
|
--ipc=host --ulimit memlock=-1:-1 \
|
|
--cap-add SYS_PTRACE \
|
|
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
|
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
|
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
|
|
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
|
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
|
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
|
|
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
|
"$FRANK_PRIMARY_IMAGE" /models \
|
|
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
|
--tensor-parallel-size 2 \
|
|
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
|
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
|
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
|
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
|
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
|
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
|
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
|
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
|
--enable-prefix-caching \
|
|
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
|
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
|
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
|
${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
|
|
${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
|
|
${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
|
|
--trust-remote-code \
|
|
> "$LOG" 2>&1
|
|
|
|
# From here on, this script's lifetime IS the container's: wait returns the
|
|
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
|
# Restart=on-failure is real. (systemd stop goes through ExecStop in the
|
|
# unit — podman wait alone cannot stop the container child.)
|
|
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|