frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
This commit is contained in:
Frank
+53
-63
@@ -1201,85 +1201,76 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the R9V radiance-vLLM fork (docker-
|
||||
# pinned prebuilt, gfx1201, driven through the docker->podman shim) serving
|
||||
# the unsloth Qwen3.8-Flash-Next Q8_0 payload (6 GGUF shards) with the MTP4
|
||||
# drafter on the two R9700s as a ROOT unit whose serve script execs the
|
||||
# fork's launcher through rootless podman as dank. Same port and alias as
|
||||
# the endpoints it replaced (the 2026-09-27 cutover from uncensored
|
||||
# IQ4_XS); the harness models.yml id is unchanged. The uncensored state dir
|
||||
# (/home/dank/r9v-state/uncensored) stays on disk as the rollback target.
|
||||
# deploy_llm_primary — frank-primary is the tcclaviger RDNA4 vLLM fork
|
||||
# (digest-pinned image, gfx1201) serving the orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8 payload (131 safetensors shards) with
|
||||
# MTP4 spec decode and q8/q8 KV on the two R9700s as a ROOT unit whose serve
|
||||
# script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`). 2026-09-29 cutover from the R9V radiance fork, whose
|
||||
# launcher/shim/setup-state machinery is retired (state dirs stay on disk as
|
||||
# rollback; pre-cutover tag frank-primary-r9v-q8-20260929). Same port and
|
||||
# alias as the endpoints it replaced; the harness models.yml id is unchanged.
|
||||
#
|
||||
# Gates: the on-host R9V setup state must exist and be ready (it carries the
|
||||
# pinned image digest and every payload path), the pinned runtime image must
|
||||
# be loadable from dank's rootless storage by its OCI manifest digest, the
|
||||
# PLE payload must match its pinned byte count, and every GGUF shard + the
|
||||
# mmproj must exist (per-file sha256 verification was setup's job at fetch
|
||||
# time, never re-hashed here).
|
||||
LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-r9v
|
||||
# Gates: the pinned runtime image must be loadable from dank's rootless
|
||||
# storage by its OCI manifest digest (the env file's FRANK_PRIMARY_IMAGE),
|
||||
# the payload dir must carry config.json and the expected shard count
|
||||
# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
|
||||
# sha256 verification was the fetch script's job at download time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
deploy_llm_primary() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
if [[ ! -f "$LLM_PRIMARY_STATE" ]]; then
|
||||
echo "ERROR: R9V setup state absent: $LLM_PRIMARY_STATE" >&2
|
||||
echo "frank-primary: run the profile setup on this host first" >&2
|
||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
||||
local image model_dir shards count
|
||||
if [[ ! -f "$env_file" ]]; then
|
||||
echo "ERROR: frank-primary env file absent: $env_file" >&2
|
||||
return 1
|
||||
fi
|
||||
local state_json image_digest model_dir ple_path ple_bytes
|
||||
state_exports="$(python3 - "$LLM_PRIMARY_STATE" <<'PY'
|
||||
import json, shlex, sys
|
||||
state = json.load(open(sys.argv[1]))
|
||||
config = state.get("config") or {}
|
||||
if state.get("ready") is not True:
|
||||
sys.exit("R9V setup state is not ready")
|
||||
for key in ("R9V_IMAGE", "R9V_MODEL_DIR", "R9V_PLE_PATH", "R9V_PLE_EXPECTED_BYTES",
|
||||
"R9V_TARGET_REL", "R9V_TARGET_SHARD2_REL", "R9V_TARGET_SHARD3_REL",
|
||||
"R9V_TARGET_SHARD4_REL", "R9V_TARGET_SHARD5_REL", "R9V_TARGET_SHARD6_REL",
|
||||
"R9V_MMPROJ_REL"):
|
||||
print(f"{key}={shlex.quote(config.get(key, ''))}")
|
||||
PY
|
||||
)" || return 1
|
||||
eval "$state_exports"
|
||||
if [[ -z "$R9V_IMAGE" || -z "$R9V_MODEL_DIR" || -z "$R9V_PLE_PATH" ]]; then
|
||||
echo "ERROR: R9V setup state is missing image/model/PLE paths" >&2
|
||||
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
|
||||
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
|
||||
shards="$(sed -n 's/^FRANK_PRIMARY_MODEL_SHARDS=//p' "$env_file" | tail -1)"
|
||||
if [[ -z "$image" || -z "$model_dir" || -z "$shards" ]]; then
|
||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_SHARDS" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
|
||||
echo "ERROR: pinned R9V runtime image absent from rootless storage: $R9V_IMAGE" >&2
|
||||
echo "frank-primary: load the image bundle (./r9v setup ...)" >&2
|
||||
if ! podman images --format '{{.RepoDigests}}' | grep -qF "${image#*@}"; then
|
||||
echo "ERROR: pinned runtime image absent from rootless storage: $image" >&2
|
||||
echo "frank-primary: fetch it (podman pull $image)" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ -n "$R9V_PLE_EXPECTED_BYTES" ]]; then
|
||||
ple_bytes="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
|
||||
if [[ "$ple_bytes" != "$R9V_PLE_EXPECTED_BYTES" ]]; then
|
||||
echo "ERROR: PLE payload wrong size: $R9V_PLE_PATH (expected $R9V_PLE_EXPECTED_BYTES, found $ple_bytes)" >&2
|
||||
echo "frank-primary: re-run the PLE derivation (./r9v setup ...)" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/config.json" ]]; then
|
||||
echo "ERROR: payload config missing: $model_dir/config.json" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/model.safetensors.index.json" ]]; then
|
||||
echo "ERROR: weight-map index missing: $model_dir/model.safetensors.index.json" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
count="$(find "$model_dir" -maxdepth 1 -name 'model-*-of-*.safetensors' 2>/dev/null | wc -l)"
|
||||
if [[ "$count" != "$shards" ]]; then
|
||||
echo "ERROR: payload shard count wrong: $model_dir (found $count, want $shards)" >&2
|
||||
echo "frank-primary: re-run scripts/download-frank-primary.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
local rel
|
||||
for rel in "$R9V_TARGET_REL" "${R9V_TARGET_SHARD2_REL:-}" "${R9V_TARGET_SHARD3_REL:-}" \
|
||||
"${R9V_TARGET_SHARD4_REL:-}" "${R9V_TARGET_SHARD5_REL:-}" "${R9V_TARGET_SHARD6_REL:-}" \
|
||||
"$R9V_MMPROJ_REL"; do
|
||||
[[ -z "$rel" ]] && continue
|
||||
if [[ ! -f "$R9V_MODEL_DIR/$rel" ]]; then
|
||||
echo "ERROR: payload file missing: $R9V_MODEL_DIR/$rel" >&2
|
||||
echo "frank-primary: re-run the profile setup to fetch/verify the payload" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
# A stale container with the same name makes `podman run --name` fail at
|
||||
# start; remove any leftover from a failed previous attempt.
|
||||
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
|
||||
# A stopped unit can leave its container running: systemd's cgroup kill
|
||||
# races conmon's --rm cleanup, and the orphan holds the port and GPUs.
|
||||
# Stop the unit, then remove the container BEFORE waiting on the port.
|
||||
# Stop the unit, then remove the containers BEFORE waiting on the port.
|
||||
# The legacy identities (the retired R9V container and the ad hoc
|
||||
# experiment container qwen-fp8) hold the same GPUs and die here too.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
|
||||
local ctr
|
||||
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
|
||||
podman rm -f "$ctr" >/dev/null 2>&1 || true
|
||||
done
|
||||
# Nothing starts until the port is released: the lucebox cutover
|
||||
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
@@ -1288,9 +1279,8 @@ PY
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-primary.service
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
# Cold start: the launcher's preflight doctor plus payload page-in
|
||||
# (~93 GiB cold; expert cache fills after /health). Parity with every
|
||||
# predecessor on this port.
|
||||
# Cold start: FP8 page-in plus torch compile/graph capture (~15 min cold,
|
||||
# measured 2026-09-29). Parity with every predecessor on this port.
|
||||
wait_for_health frank-primary http://127.0.0.1:7770/health 1800 || return 1
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
# port cannot tell WHO answers — a stray probe holding :7770 satisfies
|
||||
|
||||
Reference in New Issue
Block a user