Frank
bd5c28c4e9
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201 built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated --no-deps install per docs/install_amd.md). Shape (first-boot conservative): ONE R9700 (second card idle; --tensor-parallel-size exists for future 2-card tuning), KV pool --num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token harness worst case and one native 262,144 conversation), MoE LRU --moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens 36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml flips reasoningContentField on the harness side). Experts + PLE pin ~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity. Container identity renamed to the unit name (Frank's call): frank-primary; vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image gate becomes podman image exists (a local build carries no RepoDigests; the old digest grep refused exactly this shape). Rollback tag: frank-primary-vllm-shapeG-20261003.
85 lines
4.9 KiB
Bash
85 lines
4.9 KiB
Bash
# frank-primary.env — FreeToken (PR #587 ROCm) serving the orcarouter
|
|
# Qwen3.8-Flash-Next-Uncensored-FP8 payload on dankserver :7770.
|
|
#
|
|
# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork (git tag
|
|
# frank-primary-vllm-shapeG-20261003 is the pre-cutover state; the 362-line
|
|
# vLLM-era env is in that tag). The engine is FreeToken at PR head
|
|
# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214), built
|
|
# on the host into a local image by
|
|
# scripts/docker/frank-primary-freetoken.Dockerfile — the PR's own
|
|
# docs/install_amd.md stack (ROCm 7.14 base image, torch 2.11.0+rocm7.14,
|
|
# triton 3.7) for gfx1201.
|
|
#
|
|
# WHY FREETOKEN: the vLLM fork's expert offload planned VRAM/host tiers per
|
|
# boot from an estimator (util 0.982 law, arena 17.77 GiB / 7,762 slots,
|
|
# ~40k-token GDN state faults) and 1.0 was unbootable. FreeToken keeps ALL
|
|
# expert weights in pinned HOST RAM (~120 GiB) with an LRU GPU cache, sizes
|
|
# KV from an explicit token pool, and serves on ONE R9700 (32 GB) with the
|
|
# second card idle. RAM fit: 246 GiB total, service stopped frees the vLLM
|
|
# container's ~170 GiB; weights ~120 GiB + PLE 47.7 GiB + overhead ≈ 190 GiB.
|
|
#
|
|
# PAYLOAD: unchanged — /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8,
|
|
# 131 safetensors shards. Quant is standard modelopt block-FP8: 75,264
|
|
# weight_scale_inv tensors (block 128x128), zero input_scale (FreeToken #462
|
|
# covers exactly this export shape), one weight_scale on the PLE ngram
|
|
# embedding. FreeToken loads HF safetensors directly; no FTW conversion.
|
|
#
|
|
# SERVING FACTS (docs/cli.md, verified against main+PR587):
|
|
# - `ft serve` loads the checkpoint directly; `--served-model-name` keeps the
|
|
# harness id; reasoning/tool parsers are AUTO (reasoning lands in
|
|
# reasoning_content — models.yml flips reasoningContentField to match).
|
|
# - KV: --page-size must stay 1 for GDN/SWA models (default); --cache-type
|
|
# radix auto-picks the GDN-aware variant.
|
|
# - MoE: --moe-strategy auto -> offload (fp8 experts CANNOT use
|
|
# --moe-cpu-layers; only bf16/nvfp4/mxfp4 have a CPU executor).
|
|
# --moe-cache-rate holds the GPU LRU fraction of all experts.
|
|
# - `ft ctl cache --kv --moe` grows the pools LIVE on a running server
|
|
# (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
|
|
# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
|
|
# x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
|
|
# not per-token KV). FRANK_PRIMARY_FT_NUM_TOKENS=300000 -> ~7.2 GiB.
|
|
# Harness worst case = 140,492-token prompt + 36,864 max_tokens = 177,356
|
|
# < 300,000; one NATIVE 262,144-token conversation + full budget also fits.
|
|
# - PLE: the ~47.7 GiB ngram-embedding table pins in HOST RAM by default
|
|
# (--ple-backend disk is the alternative); the unit's LimitMEMLOCK=infinity
|
|
# makes the pin legal. TENS of GiB of pinned pages are why this stays a
|
|
# ROOT unit — see frank-primary.service.
|
|
|
|
# --- Identity (deploy + serve contract) --------------------------------
|
|
FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
# name across systemctl and podman ps. The vLLM-era name frank-primary-qwen
|
|
# moved into the legacy list so every deploy cleans it up.
|
|
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
|
|
# --- FreeToken engine knobs (first-boot conservative, resize live) -----
|
|
# Single GPU: the R9700 at index 0. HIP_VISIBLE_DEVICES=0 hides card 1 so
|
|
# the in-container index 0 IS the right card; --gpu 0 selects it.
|
|
FRANK_PRIMARY_FT_GPU=0
|
|
# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
|
|
# 300,000 tokens x 24 KiB = 7.2 GiB; clears the harness worst case 177,356.
|
|
# Grows live with `ft ctl cache --kv <n>`; shrink needs a restart.
|
|
FRANK_PRIMARY_FT_NUM_TOKENS=300000
|
|
# Concurrent decode slots. 2 is the first-boot conservative value; the
|
|
# radix pool admits ~2 full-length 200,704-token conversations. Raise with
|
|
# --max-running-requests only after a real-traffic look at slot occupancy.
|
|
FRANK_PRIMARY_FT_MAX_RUNNING=2
|
|
# Per-request output budget ceiling: the harness's maxTokens (36,864).
|
|
# Requests that omit max_tokens get this default instead of the engine's
|
|
# 32,768.
|
|
FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864
|
|
# GPU expert LRU cache as a FRACTION of all experts (~120 GiB fp8 total ->
|
|
# ~12 GiB VRAM at 0.10). First-boot conservative; resize live with
|
|
# `ft ctl cache --moe` after reading the boot ledger + ft ctl stats.
|
|
FRANK_PRIMARY_FT_MOE_RATE=0.10
|
|
# Escape hatch for boot-ladder experiments (YaRN, --max-prefill-length,
|
|
# --moe-cpu-threads, --sampling-defaults none, ...). Unset -> no extra
|
|
# argv. Keep multi-flag values as one string; the serve script expands it
|
|
# unquoted on purpose.
|
|
FRANK_PRIMARY_FT_EXTRA_ARGS=
|