Frank
0ff016a6b2
Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
83 lines
4.8 KiB
Bash
83 lines
4.8 KiB
Bash
# frank-primary.env — FreeToken (PR #587 ROCm) serving the orcarouter
|
|
# Qwen3.8-Flash-Next-Uncensored-FP8 payload on dankserver :7770.
|
|
#
|
|
# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork (git tag
|
|
# frank-primary-vllm-shapeG-20261003 is the pre-cutover state; the 362-line
|
|
# vLLM-era env is in that tag). The engine is FreeToken at PR head
|
|
# 77a7201f98f00d93f618c6cf8b81ffe8360e0899 (branch rocm714_torch214), built
|
|
# on the host into a local image by
|
|
# scripts/docker/frank-primary-freetoken.Dockerfile — the PR's own
|
|
# docs/install_amd.md stack (ROCm 7.14 base image, torch 2.11.0+rocm7.14,
|
|
# triton 3.7) for gfx1201.
|
|
#
|
|
# WHY FREETOKEN: the vLLM fork's expert offload planned VRAM/host tiers per
|
|
# boot from an estimator (util 0.982 law, arena 17.77 GiB / 7,762 slots,
|
|
# ~40k-token GDN state faults) and 1.0 was unbootable. FreeToken keeps ALL
|
|
# expert weights in pinned HOST RAM (~120 GiB) with an LRU GPU cache, sizes
|
|
# KV from an explicit token pool, and serves on ONE R9700 (32 GB) with the
|
|
# second card idle. RAM fit: 246 GiB total, service stopped frees the vLLM
|
|
# container's ~170 GiB; weights ~120 GiB + PLE 47.7 GiB + overhead ≈ 190 GiB.
|
|
#
|
|
# PAYLOAD: unchanged — /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8,
|
|
# 131 safetensors shards. Quant is standard modelopt block-FP8: 75,264
|
|
# weight_scale_inv tensors (block 128x128), zero input_scale (FreeToken #462
|
|
# covers exactly this export shape), one weight_scale on the PLE ngram
|
|
# embedding. FreeToken loads HF safetensors directly; no FTW conversion.
|
|
#
|
|
# SERVING FACTS (docs/cli.md, verified against main+PR587):
|
|
# - `ft serve` loads the checkpoint directly; `--served-model-name` keeps the
|
|
# harness id; reasoning/tool parsers are AUTO (reasoning lands in
|
|
# reasoning_content — models.yml flips reasoningContentField to match).
|
|
# - KV: --page-size must stay 1 for GDN/SWA models (default); --cache-type
|
|
# radix auto-picks the GDN-aware variant.
|
|
# - MoE: --moe-strategy auto -> offload (fp8 experts CANNOT use
|
|
# --moe-cpu-layers; only bf16/nvfp4/mxfp4 have a CPU executor).
|
|
# --moe-cache-rate holds the GPU LRU fraction of all experts.
|
|
# - `ft ctl cache --kv --moe` grows the pools LIVE on a running server
|
|
# (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
|
|
# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
|
|
# x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
|
|
# not per-token KV). See FRANK_PRIMARY_FT_NUM_TOKENS below for the pool
|
|
# size and its measured page-size law.
|
|
FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
# name across systemctl and podman ps. The vLLM-era name frank-primary-qwen
|
|
# moved into the legacy list so every deploy cleans it up.
|
|
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
|
|
# --- FreeToken engine knobs (first-boot conservative, resize live) -----
|
|
# Single GPU: the R9700 at index 0. HIP_VISIBLE_DEVICES=0 hides card 1 so
|
|
# the in-container index 0 IS the right card; --gpu 0 selects it.
|
|
FRANK_PRIMARY_FT_GPU=0
|
|
# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
|
|
# 299,968 tokens x 24 KiB = 7.0 GiB; clears the harness worst case
|
|
# (177,356 tokens) and one NATIVE 262,144-token conversation + full budget.
|
|
# MEASURED FIRST BOOT (2026-10-05): the auto attention backend is
|
|
# qsa_sparse, which OVERRIDES page size to 64, so --num-tokens must be a
|
|
# MULTIPLE OF 64 — 300,000 was refused ("nearest valid values: 299968 or
|
|
# 300032"). Grows live with `ft ctl cache --kv <n>`; shrink needs a
|
|
# restart.
|
|
FRANK_PRIMARY_FT_NUM_TOKENS=299968
|
|
# Concurrent decode slots. 2 is the first-boot conservative value; the
|
|
# radix pool admits ~2 full-length 200,704-token conversations. Raise with
|
|
# --max-running-requests only after a real-traffic look at slot occupancy.
|
|
FRANK_PRIMARY_FT_MAX_RUNNING=2
|
|
# Per-request output budget ceiling: the harness's maxTokens (36,864).
|
|
# Requests that omit max_tokens get this default instead of the engine's
|
|
# 32,768.
|
|
FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864
|
|
# GPU expert LRU cache as a FRACTION of all experts (~120 GiB fp8 total ->
|
|
# ~12 GiB VRAM at 0.10). First-boot conservative; resize live with
|
|
# `ft ctl cache --moe` after reading the boot ledger + ft ctl stats.
|
|
FRANK_PRIMARY_FT_MOE_RATE=0.10
|
|
# Escape hatch for boot-ladder experiments (YaRN, --max-prefill-length,
|
|
# --moe-cpu-threads, --sampling-defaults none, ...). Unset -> no extra
|
|
# argv. Keep multi-flag values as one string; the serve script expands it
|
|
# unquoted on purpose.
|
|
FRANK_PRIMARY_FT_EXTRA_ARGS=
|