frank-primary: FreeToken (PR587 ROCm) cutover — trio, seam, deployer

Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201 built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated --no-deps install per docs/install_amd.md). Shape (first-boot conservative): ONE R9700 (second card idle; --tensor-parallel-size exists for future 2-card tuning), KV pool --num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token harness worst case and one native 262,144 conversation), MoE LRU --moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens 36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml flips reasoningContentField on the harness side). Experts + PLE pin ~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity. Container identity renamed to the unit name (Frank's call): frank-primary; vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image gate becomes podman image exists (a local build carries no RepoDigests; the old digest grep refused exactly this shape). Rollback tag: frank-primary-vllm-shapeG-20261003.
2026-10-05 19:17:13 -07:00
1 parent 4a02dc0980
commit bd5c28c4e9
@@ -1,58 +1,31 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
-# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
-# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
-# frank-primary-r9v-q8-20260929 is the pre-cutover state).
-#
-# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
-# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
-# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
-# a 104-key on-host setup state; this stack is fully expressed by flags, and
-# every flag here was qualified by boot probe on 2026-09-29 — see
-# frank-primary.env for the qualification record).
-#
-# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
-# slots, fp8 KV, KV_BYTES
-# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
-# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
-# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
-# CANNOT boot this model
-# (torch.fx ConstraintViolationError on query_start_loc, measured at both
-# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). Speculative decode
-# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
-# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
-# retirement of MTP compared the wrong pair — see frank-primary.env.
-# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
-# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
-# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
-# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
-# state fault — the evidence block is in frank-primary.env), expert prefetch
-# OFF (36% precision measured),
-# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
-# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
-# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
-# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
-# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
-# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
-# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
-# git tag frank-primary-fp8-mtp-20261001.
-#
-# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
-# the image ships no /root/.cache at all, our first boot writes 1.1 GB of vLLM
-# AOT/torch_compile artifacts plus 229 MB of comgr JIT kernels into the
-# container's ephemeral layer, and `podman rm -f` deletes them — so every
-# deploy/restart/reboot re-pays `torch.compile took 181.20 s in total` (173.19 s
-# of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache makes all
-# later boots load the compiled artifact instead, and hides nothing. PROVEN
-# LIVE 2026-10-02 on consecutive boots of this shape: cold `torch.compile took
-# 188.26 s`, warm `Directly load the compiled graph(s) for compile range
-# (1, 1024) from the cache, took 2.143 s` / `torch.compile took 3.97 s in
-# total` / `init engine … took 41.13 s (compilation: 3.98 s)`.
+# serve-frank-primary.sh — frank-primary endpoint (FreeToken PR #587 ROCm
+# build, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 gfx1201,
+# dankserver :7770; 2026-10-05 cutover from the tcclaviger vLLM fork — git
+# tag frank-primary-vllm-shapeG-20261003 is the pre-cutover state).
 #
+# The engine is driven by `podman run` DIRECTLY and the run line below IS
+# the engine argv: the image carries FreeToken installed at pinned PR head
+# 77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile builds it from
+# the PR's own docs/install_amd.md stack). Unlike the vLLM fork there is no
+# per-boot planner: experts live in pinned host RAM with an LRU GPU cache,
+# KV is an explicit token pool, and the boot ledger in the log names
+# Shape: 1 GPU (first-boot conservative — the second R9700 sits idle;
+# --tensor-parallel-size EXISTS in FreeToken, so a 2-card TP shape is future
+# tuning once this single-card shape has a decode baseline), KV pool 300,000
+# tokens (24 KiB/token f16 across the 12 full-attention layers = 7.2 GiB;
+# clears the harness worst case of 177,356 tokens and admits one NATIVE
+# 262,144-token conversation), MoE LRU at 10% of all experts (~12 GiB),
+# 2 decode slots. First-boot conservative; `ft ctl cache --kv/--moe` grows
+# pools live, and the env comment block carries the resize law. Flag
+# surface verified against docs/cli.md of the pinned head AND against
+# `ft serve --help` inside the built image; every value here is env-driven
+# (frank-primary.env).
+
 # The root unit (scripts/frank-primary.service, User=dank,
-# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
-# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
-# also works, but the unit is the fleet-parity boot-persistent shape.
+# LimitMEMLOCK=infinity) grants the pin budget: ~120 GiB of expert weights
+# plus the ~47.7 GiB PLE table pin in host RAM. A plain shell session also
+# works, but the unit is the fleet-parity boot-persistent shape.
 set -euo pipefail

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -67,27 +40,30 @@ set -a
 source "$ENV_FILE"
 set +a

-: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the digest-pinned image}"
+: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the FreeToken image}"
 : "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the FP8 payload}"
 : "${FRANK_PRIMARY_MODEL_SHARDS:?FRANK_PRIMARY_MODEL_SHARDS must give the shard count}"
 : "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
 : "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
 : "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
-: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
-: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
-: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
-: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (auto = bf16 KV)}"
-: "${FRANK_PRIMARY_REASONING_PARSER:?FRANK_PRIMARY_REASONING_PARSER must be set (qwen3)}"
-: "${FRANK_PRIMARY_TOOL_PARSER:?FRANK_PRIMARY_TOOL_PARSER must be set (qwen3_xml)}"
-: "${FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS:?FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS must be set}"
-# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
-# classic `podman images | grep -q` pattern races — grep exits on its first
-# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
-# turns the 141 into "no match" (the 2026-09-30 deploy refusal).
-DIGESTS="$(podman images --format '{{.RepoDigests}}')"
-if ! grep -qF "${FRANK_PRIMARY_IMAGE#*@}" <<<"$DIGESTS"; then
-    echo "serve-frank-primary: pinned image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
-    echo "  fetch it: podman pull $FRANK_PRIMARY_IMAGE" >&2
+: "${FRANK_PRIMARY_FT_GPU:?FRANK_PRIMARY_FT_GPU must name the GPU index}"
+: "${FRANK_PRIMARY_FT_NUM_TOKENS:?FRANK_PRIMARY_FT_NUM_TOKENS must size the KV pool}"
+: "${FRANK_PRIMARY_FT_MAX_RUNNING:?FRANK_PRIMARY_FT_MAX_RUNNING must be set}"
+: "${FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS:?FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS must be set}"
+: "${FRANK_PRIMARY_FT_MOE_RATE:?FRANK_PRIMARY_FT_MOE_RATE must be set}"
+
+# The image gate is EXISTENCE IN STORAGE, not a registry digest: the tag is
+# built ON THE HOST (podman build from
+# scripts/docker/frank-primary-freetoken.Dockerfile) and local builds carry
+# no RepoDigests entry — the 362-line vLLM-era RepoDigests grep refused
+# exactly this shape. Pinning lives in the Dockerfile instead: the FreeToken
+# source is checked out at a hardcoded PR-head SHA and the base image is
+# digest-pinned there. `podman image exists` accepts both a local tag and a
+# name@digest ref, and it is one command — no pipefail/grep race (the
+# 2026-09-30 lesson) for either shape.
+if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
+    echo "serve-frank-primary: image absent from rootless storage: $FRANK_PRIMARY_IMAGE" >&2
+    echo "  build it: podman build -t $FRANK_PRIMARY_IMAGE -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/" >&2
    exit 1
 fi

@@ -114,8 +90,8 @@ fi

 # A crashed/killed unit can leave the container name registered (systemd
 # SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
-# identities (the retired R9V container and the ad hoc experiment container
-# both hold these GPUs); all idempotent.
+# identities (the vLLM-era frank-primary-qwen, the retired R9V container and
+# the ad hoc experiment containers all hold this GPU); all idempotent.
 for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
    podman rm -f "$CTR" >/dev/null 2>&1 || true
 done
@@ -123,74 +99,34 @@ done
 LOG=/home/dank/logs/frank_primary.log
 mkdir -p "$(dirname "$LOG")"

-# Persistent compile cache (see the header). Create it before the mount so even
-# the first, cold boot has a target; unset -> no mount, exactly the old
-# ephemeral behaviour. It carries /root/.cache/vllm (AOT + torch_compile) and
-# /root/.cache/comgr (HIP kernel JIT).
-CACHE_DIR="${FRANK_PRIMARY_CACHE_DIR:-}"
-if [ -n "$CACHE_DIR" ]; then
-    mkdir -p "$CACHE_DIR"
-fi
-
 # --device /dev/dri is the WHOLE DIRECTORY (individual renderD nodes enumerate
-# without the card nodes and HIP refuses contexts — "No CUDA GPUs are
-# available" while device_count() says 2). --ipc=host for the engine's shared
-# memory; memlock unbounded for the pinned host slabs.
-# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
-# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
-# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
-# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
-# REVERTED the same day (live conversations still degraded at ~40k with
-# caching off and zero faults — see the evidence block in
-# frank-primary.env). NO comment lines inside the backslash chain below —
-# a comment there terminates the continuation and the next line runs as
-# its own command (the 2026-10-05 exit-127 crash loop).
-# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
-# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
-# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
-# is branched explicitly: exactly one of the two reaches the engine.
-if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
-else
-    FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
-fi
+# without the card nodes and HIP refuses contexts). --ipc=host for the
+# engine's shared memory; memlock unbounded for the ~120 GiB pinned expert
+# rows + the 47.7 GiB PLE pin (the unit's LimitMEMLOCK=infinity is the other
+# half of that law).
+# HIP_VISIBLE_DEVICES pins visibility to ONE card so in-container index 0 is
+# the right R9700; --gpu 0 selects it inside the visible set.
 podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
  --publish "$FRANK_PRIMARY_PUBLISH" \
  --ipc=host --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE \
  --device /dev/kfd --device /dev/dri --device /dev/mem \
-  -e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-  ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
-  ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
-  ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
-  ${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
-  ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
+  -e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU" \
  -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-  "$FRANK_PRIMARY_IMAGE" /models \
-  --served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
-  --tensor-parallel-size 2 \
-  ${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
-  --max-model-len "$FRANK_PRIMARY_MAXLEN" \
-  --max-num-seqs "$FRANK_PRIMARY_SEQS" \
-  ${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
-  ${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory} \
-  ${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"} \
-  --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
-  --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
-  --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  "$FRANK_PRIMARY_PREFIX_CACHE_FLAG" \
-  --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
-  --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
-  --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
-  ${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"} \
-  ${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"} \
-  ${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"} \
-  --trust-remote-code \
+  "$FRANK_PRIMARY_IMAGE" \
+  ft serve --model /models \
+    --served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
+    --host 0.0.0.0 --port 7770 \
+    --gpu "$FRANK_PRIMARY_FT_GPU" \
+    --num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS" \
+    --max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING" \
+    --max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS" \
+    --moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE" \
+    ${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS} \
  > "$LOG" 2>&1

 # From here on, this script's lifetime IS the container's: wait returns the
 # container's exit code, so a crash is a nonzero ExecStart and the unit's
-# Restart=on-failure is real. (systemd stop goes through ExecStop in the
-# unit — podman wait alone cannot stop the container child.)
+# Restart=always is real. (systemd stop goes through ExecStop in the unit —
+# podman wait alone cannot stop the container child.)
 exec podman wait "$FRANK_PRIMARY_CONTAINER"