frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV

Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
2026-09-30 00:46:30 -07:00
1 parent 9cf4b1de82
commit 3bb0e82ccf
@@ -1,30 +1,27 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (R9V radiance-vLLM fork,
-# unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 drafter, 2x R9700, dankserver
-# :7770; 2026-09-27 cutover from the uncensored IQ4_XS payload, whose state
-# dir stays on disk as the rollback target).
+# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
+# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
+# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
+# frank-primary-r9v-q8-20260929 is the pre-cutover state).
 #
-# The engine is the upstream R9V launcher (scripts/launch.sh in the fork
-# checkout), NOT a hand-frozen podman argv: launch.sh owns the resolved
-# docker/podman args (binds, overlays, device passthrough, publish lines)
-# and the fork's qualification was measured through it. This script's job:
+# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
+# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
+# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
+# a 104-key on-host setup state; this stack is fully expressed by flags, and
+# every flag here was qualified by boot probe on 2026-09-29 — see
+# frank-primary.env for the qualification record).
 #
-#   1. layer the on-host R9V setup state (the 104-key tuned shape the
-#      profile setup qualified) under this repo's frank-primary.env
-#      overrides (bind/port/alias/container);
-#   2. gate on the pinned runtime image (by its OCI manifest digest, the
-#      podman RepoDigest), the exact-size PLE payload, and the GGUF shards;
-#   3. exec the launcher, then wait on the container: launch.sh --detach
-#      exits as soon as the container is created, so without the wait the
-#      unit would be "green" while the engine crashes — the wait is what
-#      keeps systemd's Restart=on-failure self-healing real.
+# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
+# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
+# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
+# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
+# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
+# precision — the tuned incumbent deletes it), TunableOp off.
 #
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert UVA
-# host copy and the PLE pinned residency (54.4 GB on the q8 payload). A
-# plain shell session also works (proven 2026-09-27: podman rootless
-# inherits usable limits here), but the unit is the fleet-parity
-# boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
+# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
+# also works, but the unit is the fleet-parity boot-persistent shape.
 set -euo pipefail

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -39,104 +36,85 @@ set -a
 source "$ENV_FILE"
 set +a

-: "${R9V_HOME:?R9V_HOME must name the fork checkout}"
-: "${R9V_STATE_FILE:?R9V_STATE_FILE must name the setup state}"
-: "${R9V_HOST_BIND:?R9V_HOST_BIND must be set}"
-: "${R9V_HOST_PORT:?R9V_HOST_PORT must be set}"
-: "${R9V_SERVED_MODEL_NAME:?R9V_SERVED_MODEL_NAME must be set}"
-: "${R9V_CONTAINER_NAME:?R9V_CONTAINER_NAME must be set}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
+: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
+: "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
+: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
+: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
+: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
+: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
+: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
+: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
+: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
+: "${FRANK_PRIMARY_SPEC:?FRANK_PRIMARY_SPEC must be set (0 disables)}"
+: "${FRANK_PRIMARY_UTIL:?FRANK_PRIMARY_UTIL must be set}"

-LAUNCHER="$R9V_HOME/scripts/launch.sh"
-if [ ! -x "$LAUNCHER" ]; then
-    echo "serve-frank-primary: R9V launcher missing: $LAUNCHER" >&2
+# --- gates: never serve a half-finished payload or a missing image -----------
+if ! podman images --format '{{.RepoDigests}}' | grep -qF "${FRANK_PRIMARY_IMAGE#*@}"; then
+    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
+    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
    exit 1
 fi

-# The docker->podman shim is REQUIRED: the launcher and its doctor speak
-# docker (OCI-manifest-digest image IDs, LogConfig fields) that podman's
-# native CLI answers differently. The shim carries those laws.
-SHIM_DIR="$R9V_HOME/.podman-shim"
-if [ ! -x "$SHIM_DIR/docker" ]; then
-    echo "serve-frank-primary: docker->podman shim missing: $SHIM_DIR/docker" >&2
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/config.json" ]; then
+    echo "serve-frank-primary: payload config missing: $FRANK_PRIMARY_MODEL_DIR/config.json" >&2
    exit 1
 fi
-export PATH="$SHIM_DIR:$PATH"
-
-# --- layer 1: the qualified setup state (production overrides never come
-# from here; keys already defined in the env file are skipped) -------------
-STATE_FILE="$R9V_STATE_FILE"
-if [ ! -f "$STATE_FILE" ]; then
-    echo "serve-frank-primary: R9V setup state missing: $STATE_FILE" >&2
-    echo "  run the profile setup first (tools/setup_profile.py setup --state-dir ...)" >&2
+SHARDS="$(find "$FRANK_PRIMARY_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
+if [ "$SHARDS" != "$FRANK_PRIMARY_MODEL_SHARDS" ]; then
+    echo "serve-frank-primary: payload shard count wrong: $FRANK_PRIMARY_MODEL_DIR" >&2
+    echo "  found $SHARDS safetensors, want $FRANK_PRIMARY_MODEL_SHARDS" >&2
+    echo "  re-fetch: scripts/download-frank-primary.sh" >&2
    exit 1
 fi
-STATE_EXPORTS="$(python3 - "$STATE_FILE" <<'PY'
-import json, shlex, sys, os
-state = json.load(open(sys.argv[1]))
-config = state.get("config") or {}
-if state.get("ready") is not True:
-    sys.exit("R9V setup state is not ready; rerun setup before serving")
-env_keys = {k for k in os.environ if k.startswith("R9V_")}
-for key in sorted(config):
-    if key.startswith("R9V_") and key not in env_keys:
-        print(f"export {key}={shlex.quote(config[key])}")
-PY
-)"
-eval "$STATE_EXPORTS"
-
-: "${R9V_IMAGE:?state must carry R9V_IMAGE}"
-: "${R9V_MODEL_DIR:?state must carry R9V_MODEL_DIR}"
-: "${R9V_PLE_PATH:?state must carry R9V_PLE_PATH}"
-
-# --- gates: never serve a half-finished payload or a missing image --------
-# Runtime image by pinned OCI manifest digest (podman keys images by config
-# digest but preserves the manifest digest in RepoDigests — the shim's ID law).
-if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
-    echo "serve-frank-primary: pinned R9V image absent: $R9V_IMAGE" >&2
-    echo "  load the pinned image bundle into podman rootless storage" >&2
+if [ ! -f "$FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" ]; then
+    echo "serve-frank-primary: weight-map index missing: $FRANK_PRIMARY_MODEL_DIR/model.safetensors.index.json" >&2
    exit 1
 fi

-# PLE payload at its pinned byte count (stat -c %s, never wc -c), plus the
-# six target GGUF shards (mradermacher-style split parts) and the mmproj
-# the launcher mounts.
-PLE_BYTES="${R9V_PLE_EXPECTED_BYTES:-}"
-if [ -n "$PLE_BYTES" ]; then
-    ACTUAL="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
-    if [ "$ACTUAL" != "$PLE_BYTES" ]; then
-        echo "serve-frank-primary: PLE payload wrong size: $R9V_PLE_PATH" >&2
-        echo "  expected $PLE_BYTES, found $ACTUAL" >&2
-        exit 1
-    fi
-else
-    if [ ! -f "$R9V_PLE_PATH" ]; then
-        echo "serve-frank-primary: PLE payload missing: $R9V_PLE_PATH" >&2
-        exit 1
-    fi
-fi
-for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
-    "${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
-    "$R9V_MMPROJ_REL"; do
-    if [ -z "$rel" ]; then continue; fi
-    if [ ! -f "$R9V_MODEL_DIR/$rel" ]; then
-        echo "serve-frank-primary: payload file missing: $R9V_MODEL_DIR/$rel" >&2
-        exit 1
-    fi
-done
-
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
    echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
    exit 1
 fi

 # A crashed/killed unit can leave the container name registered (systemd
-# SIGKILLs conmon before cleanup runs); the launcher refuses an existing
-# name forever after. Clear any orphan of OUR name first — idempotent.
-docker rm -f "$R9V_CONTAINER_NAME" >/dev/null 2>&1 || true
+# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
+# identities (the retired R9V container and the ad hoc experiment container
+# both hold these GPUs); all idempotent.
+for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
+    podman rm -f "$CTR" >/dev/null 2>&1 || true
+done

-# launch.sh detaches after preflight; from here on, this script's lifetime
-# IS the container's: wait returns the container's exit code, so a crash is
-# a nonzero ExecStart and Restart=on-failure fires. (systemd stop goes
-# through ExecStop in the unit — docker wait alone cannot stop the child.)
-"$LAUNCHER"
-exec docker wait "$R9V_CONTAINER_NAME"
+LOG=/home/dank/logs/frank_primary.log
+mkdir -p "$(dirname "$LOG")"
+
+# --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
+# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
+# available" while device_count() says 2). --ipc=host for the engine's shared
+# memory; memlock unbounded for the pinned host slabs.
+# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
+podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
+  --publish "$FRANK_PRIMARY_PUBLISH" \
+  --ipc=host --ulimit memlock=-1:-1 \
+  --cap-add SYS_PTRACE \
+  --device /dev/kfd --device /dev/dri --device /dev/mem \
+  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
+  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
+  "$FRANK_PRIMARY_IMAGE" /models \
+  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
+  --tensor-parallel-size 2 \
+  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
+  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
+  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
+  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
+  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
+  --enable-prefix-caching \
+  --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
+  --trust-remote-code \
+  > "$LOG" 2>&1
+
+# From here on, this script's lifetime IS the container's: wait returns the
+# container's exit code, so a crash is a nonzero ExecStart and the unit's
+# Restart=on-failure is real. (systemd stop goes through ExecStop in the
+# unit — podman wait alone cannot stop the container child.)
+exec podman wait "$FRANK_PRIMARY_CONTAINER"