Frank
9d8b18553d
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers each — then the process-lifetime report cap. The guard's own text in this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix caching … Mitigation: --no-enable-prefix-caching'. Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which checkpoints each sequence's GDN recurrent state into a mamba page per 3136-token block and re-seeds it from that page when a request resumes from a cached prefix. The NaN state landed at the 13th boundary (13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page: every later turn of that conversation re-seeded the same non-finite state for two heads, silently (fault-report cap already spent). RAM ruled out the same day: zero MCE/EDAC records since the boot under test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN signature is a state-path fault, not scattered bit corruption. Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching (flag renders in both directions from the knob). Cost: every turn re-prefills the whole history instead of at most 3135 tail tokens. Seam updated: the prefix-cache pin now asserts the conditional render and the shipped OFF value. Runbook carries the evidence + acceptance criteria; all 368 tests pass.
185 lines
10 KiB
Bash
Executable File
185 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
|
|
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
|
|
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
|
|
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
|
|
#
|
|
# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
|
|
# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
|
|
# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
|
|
# a 104-key on-host setup state; this stack is fully expressed by flags, and
|
|
# every flag here was qualified by boot probe on 2026-09-29 — see
|
|
# frank-primary.env for the qualification record).
|
|
#
|
|
# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
|
|
# slots, fp8 KV, KV_BYTES
|
|
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
|
|
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
|
|
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
|
|
# CANNOT boot this model
|
|
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
|
|
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
|
|
# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
|
|
# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
|
|
# retirement of MTP compared the wrong pair — see frank-primary.env.
|
|
# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
|
|
# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
|
|
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
|
|
# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
|
|
# state fault — the evidence block is in frank-primary.env), expert prefetch
|
|
# OFF (36% precision measured),
|
|
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
|
|
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
|
|
# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
|
|
# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
|
|
# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
|
|
# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
|
|
# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
|
|
# git tag frank-primary-fp8-mtp-20261001.
|
|
#
|
|
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
|
|
# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
|
|
# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
|
|
# container's ephemeral layer, and `podman rm -f` deletes them — so every
|
|
# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
|
|
# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
|
|
# later boots load the compiled artifact instead, and hides nothing. PROVEN
|
|
# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
|
|
# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
|
|
# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
|
|
# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
|
|
#
|
|
# The root unit (scripts/frank-primary.service, User=dank,
|
|
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
|
# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
|
|
# also works, but the unit is the fleet-parity boot-persistent shape.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
|
|
|
|
if [ ! -f "$ENV_FILE" ]; then
|
|
echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
|
|
exit 1
|
|
fi
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
set +a
|
|
|
|
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
|
|
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
|
|
: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
|
|
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
|
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
|
: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
|
|
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
|
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
|
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
|
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
|
|
: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
|
|
: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
|
|
: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
|
|
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
|
# classic `podman images | grep -q` pattern races — grep exits on its first
|
|
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
|
# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
|
|
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
|
if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
|
|
echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
|
|
echo " fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
|
|
echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
|
|
exit 1
|
|
fi
|
|
SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
|
if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
|
|
echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
|
|
echo " found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
|
|
echo " re-fetch: scripts/download-frank-primary.sh" >&2
|
|
exit 1
|
|
fi
|
|
if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
|
|
echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
|
echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# A crashed/killed unit can leave the container name registered (systemd
|
|
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
|
# identities (the retired R9V container and the ad hoc experiment container
|
|
# both hold these GPUs); all idempotent.
|
|
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
|
done
|
|
|
|
LOG=/home/dank/logs/frank_primary.log
|
|
mkdir -p "$(dirname "$LOG")"
|
|
|
|
# Persistent compile cache (see the header). Create it before the mount so even
|
|
# the first, cold boot has a target; unset -> no mount, exactly the old
|
|
# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
|
|
# /root/.cache/comgr (HIP kernel JIT).
|
|
CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
|
|
if [ -n "$CACHE_DIR" ]; then
|
|
mkdir -p "$CACHE_DIR"
|
|
fi
|
|
|
|
# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
|
|
# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
|
|
# available" while device_count() says 2). --ipc=host for the engine's shared
|
|
# memory; memlock unbounded for the pinned host slabs.
|
|
# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
|
|
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|
--publish "$FRANK_PRIMARY_PUBLISH" \
|
|
--ipc=host --ulimit memlock=-1:-1 \
|
|
--cap-add SYS_PTRACE \
|
|
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
|
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
|
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
|
|
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
|
|
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
|
|
${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
|
|
${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
|
|
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
|
|
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
|
"$FRANK_PRIMARY_IMAGE" /models \
|
|
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
|
--tensor-parallel-size 2 \
|
|
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
|
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
|
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
|
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
|
${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
|
|
${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
|
|
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
|
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
|
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
|
# Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state at
|
|
# ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request). FRANK_
|
|
# PRIMARY_NO_PREFIX_CACHING flips to the fork-named mitigation
|
|
# (--no-enable-prefix-caching); empty keeps the historical flag. The
|
|
# evidence block is in frank-primary.env.
|
|
${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching} \
|
|
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
|
|
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
|
|
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
|
|
${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
|
|
${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
|
|
${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
|
|
--trust-remote-code \
|
|
> "$LOG" 2>&1
|
|
|
|
# From here on, this script's lifetime IS the container's: wait returns the
|
|
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
|
# Restart=on-failure is real. (systemd stop goes through ExecStop in the
|
|
# unit — podman wait alone cannot stop the container child.)
|
|
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|