frank-primary: radiance native-FP8 TP=2 cutover (engine, payload build, rollback)
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8 payload at native block-FP8 precision: - scripts/build-frank-primary-rad.sh: staging (quantization_config drop = plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE --quant rule (PLE rtn fixed-scale); plan-only first. - scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes (experts byte-stacked into the plugin's stacked layout; PLE's 128 scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own scale; the rtn rule re-encodes at that same scale - codes round-trip exactly and land the one-element scale plane the plugin declares). - scripts/hfget.sh + verified download (131/131 shards sha256-match). - froggeric v22.5 chat template vendored (--override-chat-template). - serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host), TTM preflight fixed for kernels with pages_limit only (no pages_max - an absent knob is skipped, not read as zero); pages_limit raised to 52,428,800 pages = 200 GiB on the host pre-cutover. - root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB + health; prometheus re-adds the /metrics job. - rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile); payload + image stay on disk until the window closes. deploy-remote.sh also carries the parallel session's frank-small payload gates (swift15) - committed here per Frank's direction to commit the tree.
This commit is contained in:
Frank
+49
-56
@@ -1207,12 +1207,14 @@ deploy_llm_large() {
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
||||
LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}"
|
||||
LLM_SMALL_MODEL_FILE="${LLM_SMALL_MODEL_FILE:-/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}"
|
||||
LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# The projector IS the vision capability here: the Ornith GGUF ships no
|
||||
# `v.*` tensors, so a deploy that starts without this file silently ships a
|
||||
# The projector IS the vision capability here: no Swift repo ships one (the
|
||||
# GGUF carries no vision tensors), so the file is unsloth's mmproj built
|
||||
# from the shared Qwen3.8-27B base — proven byte-identical to Swift's own
|
||||
# vision stack. A deploy that starts without this file silently ships a
|
||||
# text-only endpoint. Checked unconditionally, like frank-large's.
|
||||
LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}"
|
||||
LLM_SMALL_MMPROJ="${LLM_SMALL_MMPROJ:-/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf}"
|
||||
|
||||
deploy_llm_small() {
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
@@ -1250,23 +1252,24 @@ deploy_llm_small() {
|
||||
echo "frank-small: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the FreeToken engine (PR #587 ROCm
|
||||
# build, gfx1201) serving the orcarouter Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
# payload (131 safetensors shards) on ONE R9700 as a ROOT unit whose serve
|
||||
# script drives `podman run` directly (the image carries `ft serve`).
|
||||
# 2026-10-05 cutover from the tcclaviger vLLM fork (pre-cutover tag
|
||||
# frank-primary-vllm-shapeG-20261003); before that, 2026-09-29 cutover from
|
||||
# the R9V radiance fork, whose launcher/shim/setup-state machinery is
|
||||
# retired. Same port and alias as the endpoints it replaced; the harness
|
||||
# models.yml id is unchanged.
|
||||
# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
|
||||
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
|
||||
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
|
||||
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
|
||||
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
||||
# the image entrypoint; the image lives in dank's rootless podman storage).
|
||||
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
||||
# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||
# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
|
||||
# harness models.yml id is unchanged.
|
||||
#
|
||||
# Gates: the runtime image must exist in dank's rootless storage
|
||||
# (FRANK_PRIMARY_IMAGE — a LOCAL tag: the image is built on the host from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile, which pins the
|
||||
# FreeToken PR-head SHA and the base image), the payload dir must carry
|
||||
# config.json and the expected shard count (FRANK_PRIMARY_MODEL_SHARDS),
|
||||
# and the unit env must be complete. Per-file sha256 verification was the
|
||||
# fetch script's job at download time, never re-hashed here.
|
||||
# Gates: the image must exist under its digest ref in dank's rootless
|
||||
# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
|
||||
# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
|
||||
# a truncated convert is a guaranteed long-timeout boot), and the unit env
|
||||
# must be complete. Per-file sha256 verification is the fetch script's job
|
||||
# at download time and rad-convert/rad-info's job at build time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
|
||||
@@ -1275,44 +1278,36 @@ deploy_llm_primary() {
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
||||
local image model_dir shards count
|
||||
local image model_dir model_file rad_min mib
|
||||
if [[ ! -f "$env_file" ]]; then
|
||||
echo "ERROR: frank-primary env file absent: $env_file" >&2
|
||||
return 1
|
||||
fi
|
||||
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
|
||||
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
|
||||
shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
|
||||
if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
|
||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
|
||||
model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
|
||||
rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
||||
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
|
||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||
return 1
|
||||
fi
|
||||
# The image gate is EXISTENCE IN STORAGE, not a registry digest: the
|
||||
# FreeToken image is built ON THE HOST (a local tag carries NO
|
||||
# RepoDigests entry — the old grep refused exactly this shape), and
|
||||
# pinning lives in the Dockerfile (pinned PR-head SHA + digest-pinned
|
||||
# base image). `podman image exists` accepts a local tag and a
|
||||
# name@digest ref alike, in one command — no pipefail/grep race (the
|
||||
# 2026-09-30 lesson) for either shape.
|
||||
# The image gate is the digest ref itself: a name@sha256 ref exists in
|
||||
# storage only if THAT manifest was pulled. One command — no
|
||||
# pipefail/grep race (the 2026-09-30 lesson).
|
||||
if ! podman image exists "$image"; then
|
||||
echo "ERROR: runtime image absent from rootless storage: $image" >&2
|
||||
echo "frank-primary: build it (scripts/docker/frank-primary-freetoken.Dockerfile) or pull it" >&2
|
||||
echo "ERROR: runtime image absent from storage: $image" >&2
|
||||
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/config.json" ]]; then
|
||||
echo "ERROR: payload config missing: $model_dir/config.json" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
if [[ ! -f "$model_dir/$model_file" ]]; then
|
||||
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
||||
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
|
||||
echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
|
||||
if [[ "$count" != "$shards" ]]; then
|
||||
echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
||||
if [[ "$mib" -lt "$rad_min" ]]; then
|
||||
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
||||
echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
@@ -1337,18 +1332,16 @@ deploy_llm_primary() {
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-primary.service
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
# Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
|
||||
# measured 2026-09-29). Parity with every predecessor on this port.
|
||||
# FreeToken's /health returns 200 with {"status":"loading"} while the
|
||||
# engine loads (the FastAPI front end binds before the scheduler is
|
||||
# up), so a bare 200 false-greens — measured 2026-10-06: "healthy after
|
||||
# 6 s" while 131 shards were still paging in. vLLM 503'd during load,
|
||||
# which made the bare curl honest there; FreeToken needs the BODY read.
|
||||
# Only status:"ok" counts. (The MainPID check below still guards the
|
||||
# stray-port case.)
|
||||
# Cold start: .rad load + PLE pin + expert host-pool upload (the
|
||||
# publisher's compose healthcheck allows a 15-minute start_period).
|
||||
# Parity with every predecessor on this port. The HTTP listener binds
|
||||
# AFTER the full load (main.cpp: engine.start() then engine.serve()),
|
||||
# so during load the port REFUSES connections — a bare curl is honest
|
||||
# here, unlike FreeToken's 200-while-loading body. (The MainPID check
|
||||
# below still guards the stray-port case.)
|
||||
local i ok=0
|
||||
for ((i=1; i<=900; i++)); do
|
||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null | grep -q '"status":"ok"'; then
|
||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
|
||||
echo "frank-primary engine healthy after approximately $((i * 2))s"
|
||||
ok=1
|
||||
break
|
||||
@@ -1356,7 +1349,7 @@ deploy_llm_primary() {
|
||||
sleep 2
|
||||
done
|
||||
if [[ "$ok" != 1 ]]; then
|
||||
echo "FAIL: frank-primary never reached /health status ok in 1800s" >&2
|
||||
echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
|
||||
return 1
|
||||
fi
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
|
||||
Reference in New Issue
Block a user