Frank
3bb0e82ccf
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
121 lines
5.8 KiB
Bash
Executable File
121 lines
5.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
|
|
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
|
|
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
|
|
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
|
|
#
|
|
# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
|
|
# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
|
|
# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
|
|
# a 104-key on-host setup state; this stack is fully expressed by flags, and
|
|
# every flag here was qualified by boot probe on 2026-09-29 — see
|
|
# frank-primary.env for the qualification record).
|
|
#
|
|
# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
|
|
# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
|
|
# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
|
|
# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
|
|
# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
|
|
# precision — the tuned incumbent deletes it), TunableOp off.
|
|
#
|
|
# The root unit (scripts/frank-primary.service, User=dank,
|
|
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
|
# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
|
|
# also works, but the unit is the fleet-parity boot-persistent shape.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
|
|
|
|
if [ ! -f "$ENV_FILE" ]; then
|
|
echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
|
|
exit 1
|
|
fi
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
set +a
|
|
|
|
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
|
|
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
|
|
: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
|
|
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
|
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
|
: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
|
|
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
|
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
|
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
|
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
|
|
: "${FRANK_PRIMARY_SPEC:?FRANK_PRIMARY_SPEC must be set (0 disables)}"
|
|
: "${FRANK_PRIMARY_UTIL:?FRANK_PRIMARY_UTIL must be set}"
|
|
|
|
# --- gates: never serve a half-finished payload or a missing image -----------
|
|
if ! podman images --format '{{.RepoDigests}}' | grep -qF "${FRANK_PRIMARY_IMAGE#*@}"; then
|
|
echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
|
|
echo " fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
|
|
echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
|
|
exit 1
|
|
fi
|
|
SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
|
if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
|
|
echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
|
|
echo " found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
|
|
echo " re-fetch: scripts/download-frank-primary.sh" >&2
|
|
exit 1
|
|
fi
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
|
|
echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
|
echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# A crashed/killed unit can leave the container name registered (systemd
|
|
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
|
# identities (the retired R9V container and the ad hoc experiment container
|
|
# both hold these GPUs); all idempotent.
|
|
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
|
done
|
|
|
|
LOG=/home/dank/logs/frank_primary.log
|
|
mkdir -p "$(dirname "$LOG")"
|
|
|
|
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
|
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
|
# available" while device_count() says 2). --ipc=host for the engine's shared
|
|
# memory; memlock unbounded for the pinned host slabs.
|
|
# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
|
|
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|
--publish "$FRANK_PRIMARY_PUBLISH" \
|
|
--ipc=host --ulimit memlock=-1:-1 \
|
|
--cap-add SYS_PTRACE \
|
|
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
|
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
|
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
|
"$FRANK_PRIMARY_IMAGE" /models \
|
|
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
|
--tensor-parallel-size 2 \
|
|
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
|
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
|
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
|
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
|
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
|
--enable-prefix-caching \
|
|
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
|
|
--trust-remote-code \
|
|
> "$LOG" 2>&1
|
|
|
|
# From here on, this script's lifetime IS the container's: wait returns the
|
|
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
|
# Restart=on-failure is real. (systemd stop goes through ExecStop in the
|
|
# unit — podman wait alone cannot stop the container child.)
|
|
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|