frank-primary: FreeToken (PR587 ROCm) cutover — trio, seam, deployer

Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201
built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201
(scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated
--no-deps install per docs/install_amd.md).

Shape (first-boot conservative): ONE R9700 (second card idle;
--tensor-parallel-size exists for future 2-card tuning), KV pool
--num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token
harness worst case and one native 262,144 conversation), MoE LRU
--moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens
36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml
flips reasoningContentField on the harness side). Experts + PLE pin
~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity.

Container identity renamed to the unit name (Frank's call): frank-primary;
vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image
gate becomes podman image exists (a local build carries no RepoDigests;
the old digest grep refused exactly this shape). Rollback tag:
frank-primary-vllm-shapeG-20261003.
This commit is contained in:
Frank committed 2026-10-05 19:17:13 -07:00
1 parent 4a02dc0980
commit bd5c28c4e9
6 files changed
+347 -869

No files matched your search

+28 -26
View File
@@ -1250,24 +1250,25 @@ deploy_llm_small() {
echo "frank-small: health answered by unit MainPID $main_pid"
}
# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
# (digest-pinned image, gfx1201) serving the orcarouter
# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
# script drives `podman run` directly (the image entrypoint execs
# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
# alias as the endpoints it replaced; the harness models.yml id is unchanged.
# deploy_llm_primary — frank-primary is the FreeToken engine (PR #587 ROCm
# build, gfx1201) serving the orcarouter Qwen3.8-Flash-Next-Uncensored-FP8
# payload (131 safetensors shards) on ONE R9700 as a ROOT unit whose serve
# script drives `podman run` directly (the image carries `ft serve`).
# 2026-10-05 cutover from the tcclaviger vLLM fork (pre-cutover tag
# frank-primary-vllm-shapeG-20261003); before that, 2026-09-29 cutover from
# the R9V radiance fork, whose launcher/shim/setup-state machinery is
# retired. Same port and alias as the endpoints it replaced; the harness
# models.yml id is unchanged.
#
# Gates: the pinned runtime image must be loadable from dank's rootless
# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
# the payload dir must carry config.json and the expected shard count
# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
# sha256 verification was the fetch script's job at download time, never
# re-hashed here.
LLM_PRIMARY_CONTAINER=frank-primary-qwen
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
# Gates: the runtime image must exist in dank's rootless storage
# (FRANK_PRIMARY_IMAGE — a LOCAL tag: the image is built on the host from
# scripts/docker/frank-primary-freetoken.Dockerfile, which pins the
# FreeToken PR-head SHA and the base image), the payload dir must carry
# config.json and the expected shard count (FRANK_PRIMARY_MODEL_SHARDS),
# and the unit env must be complete. Per-file sha256 verification was the
# fetch script's job at download time, never re-hashed here.
LLM_PRIMARY_CONTAINER=frank-primary
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
deploy_llm_primary() {
# Refuse rather than restart-loop: a unit whose image or payload is
@@ -1286,15 +1287,16 @@ deploy_llm_primary() {
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
return 1
fi
# Capture podman's output BEFORE grepping: under set -o pipefail, the
# classic `podman images | grep -q` races (grep exits on match, podman
# dies SIGPIPE 141 writing the rest, pipefail calls it no-match — the
# 2026-09-30 deploy refusal).
local digests
digests="$(podman images --format '{{.RepoDigests}}')"
if ! grep -qF "${image#*@}" <<<"$digests"; then
echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
echo "frank-primary: fetch it (podman pull $image)" >&2
# The image gate is EXISTENCE IN STORAGE, not a registry digest: the
# FreeToken image is built ON THE HOST (a local tag carries NO
# RepoDigests entry — the old grep refused exactly this shape), and
# pinning lives in the Dockerfile (pinned PR-head SHA + digest-pinned
# base image). `podman image exists` accepts a local tag and a
# name@digest ref alike, in one command — no pipefail/grep race (the
# 2026-09-30 lesson) for either shape.
if ! podman image exists "$image"; then
echo "ERROR: runtime image absent from rootless storage: $image" >&2
echo "frank-primary: build it (scripts/docker/frank-primary-freetoken.Dockerfile) or pull it" >&2
return 1
fi
if [[ ! -f "$model_dir/config.json" ]]; then