frank-medium: serve the new R9700s, retire frank-small
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all, so the `frank-small` service is retired and `frank-medium` takes the endpoint. New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3 never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF -- a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load. Placeholder config: --split-mode tensor across both cards, 131072 context, 1 slot, f16 KV, MTP draft head on ROCm0. deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped, disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units before frank-medium starts, because `systemctl disable` has been observed to delete the unit file itself on this host (2026-08-26 incident). The retired frank-small repo files (env, unit, serve script) move to scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the deployment test seams now name frank-medium.
This commit is contained in:
1 parent
27e63359d6
commit
6085a99dd8
14 files changed
+454
-192
No files matched your search
@@ -1,77 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as
|
||||
# frank-small via FreeToken on Dankserver (RTX 3090, 60 GB RAM).
|
||||
#
|
||||
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
|
||||
# - Experts live in host RAM; an LRU cache of expert slots is resident
|
||||
# on the GPU. Misses are streamed over PCIe.
|
||||
# - Paged KV cache (radix prefix sharing). Only the 10 full-attention
|
||||
# layers page KV (~20 KiB/token); the 30 GDN/linear-attention layers
|
||||
# hold constant-size per-sequence state (mamba slots).
|
||||
# - OpenAI-compatible API (/v1/chat/completions, /v1/models).
|
||||
#
|
||||
# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 — Huihui
|
||||
# abliterated finetune of Qwen3.6-35B-A3B in the same native modelopt
|
||||
# block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert
|
||||
# layout) as the official FP8 checkpoint this service ran before; the
|
||||
# FreeToken loader path is unchanged. 40 layers (10 attention + 30
|
||||
# GDN), 256 experts / 8 used, 2 KV heads × 256 dim. ~35 GB on disk.
|
||||
# The checkpoint ships the fixed Qwen3.6 chat template; FreeToken reads
|
||||
# it from the model directory.
|
||||
#
|
||||
# Pool sizing: the GPU budget is split MoE-first between expert residency
|
||||
# and KV pages. --moe-cache-size pins the number of resident expert
|
||||
# slots; --num-tokens pins the KV pool capacity. Live resize is possible
|
||||
# via `ft ctl cache --kv N --moe M` (grows KV by shrinking MoE; shrinking
|
||||
# KV requires a restart due to the swap-during-rebuild memory cost).
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE
|
||||
# overrides that path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -----------------
|
||||
: "${MODEL_DIR:=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${FT_VENV:=/home/dank/freetoken/.venv}"
|
||||
: "${CUDA_HOME_DIR:=${FT_VENV}/lib/python3.12/site-packages/nvidia/cu13}"
|
||||
|
||||
# Pool sizing (expert residency vs KV cache tradeoff).
|
||||
: "${MOE_CACHE_SIZE:=2048}"
|
||||
: "${NUM_TOKENS:=453005}"
|
||||
: "${MAX_RUNNING_REQUESTS:=4}"
|
||||
|
||||
# Paths
|
||||
FT_BIN="${FT_VENV}/bin/ft"
|
||||
MODEL="${MODEL_DIR}"
|
||||
|
||||
if [[ ! -x "$FT_BIN" ]]; then
|
||||
echo "missing FreeToken CLI: $FT_BIN" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -d "$MODEL" ]]; then
|
||||
echo "missing model directory: $MODEL" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export PATH="${FT_VENV}/bin:${PATH}"
|
||||
export CUDA_HOME="${CUDA_HOME_DIR}"
|
||||
|
||||
exec "$FT_BIN" serve \
|
||||
--model "$MODEL" \
|
||||
--served-model-name "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--moe-cache-size "$MOE_CACHE_SIZE" \
|
||||
--num-tokens "$NUM_TOKENS" \
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
Reference in new issue
Block a user