Frank
306ff32427
The first deploy refused: floor 178000 > actual 177109 MiB - the plan-only estimate (172.76 GiB) prices the data planes but not the container's metadata tail. Floor 177000 sits just under the real payload.
138 lines
7.9 KiB
Bash
138 lines
7.9 KiB
Bash
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
|
|
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
|
|
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
|
|
# dankserver :7770.
|
|
#
|
|
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
|
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
|
|
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
|
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
|
# frank-primary-vllm-shapeG-20261003).
|
|
#
|
|
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
|
# gfx1201, drives `podman run` directly from the digest-pinned
|
|
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
|
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
|
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
|
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
|
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
|
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
|
# --host-pool-mib 12288) remains the documented fallback if the native
|
|
# container proves unbootable.
|
|
#
|
|
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
|
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
|
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
|
# staging dir whose config.json drops `quantization_config`: the descriptor
|
|
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
|
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
|
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
|
# containers carry. Per-weight encodings are data, not the descriptor —
|
|
# removing the key changes nothing about what the weights hold. Built by
|
|
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
|
|
#
|
|
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
|
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
|
# then engine.serve()) — during load the port refuses connections, so a
|
|
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
|
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
|
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
|
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
|
# the chat format. Both need no parser flags.
|
|
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
|
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
|
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
|
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
|
# models.yml's extraBody set is fully inside it. Unknown params are
|
|
# refused (400), which is why nothing unverified rides the wire.
|
|
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
|
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
|
# publisher's throughput peak for this model. Stated explicitly with
|
|
# --num-speculative-tokens 3.
|
|
# - The fused TP all-reduce is declared over max-num-seqs x
|
|
# (1 + --num-speculative-tokens) rows; 2 x 4 = 8 rows fits either wire.
|
|
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
|
# The .rad container FILE (mounted under /models) and the state dir for the
|
|
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
|
# .rad files; the raw HF download and its staging dir live beside it and
|
|
# are never mounted.
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
|
|
FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad
|
|
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
|
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
# name across systemctl and podman ps. The runtime changes; the name does
|
|
# not. The legacy list retires every predecessor identity that could hold
|
|
# the GPUs or the port.
|
|
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PORT=7770
|
|
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
|
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
|
# template is baked into the .rad by rad-convert and this overrides it).
|
|
# Radiance derives the reasoning/tool-call reply format from it.
|
|
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
|
|
|
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
|
|
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
|
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
|
# wire is libr4d's own, not RCCL.
|
|
FRANK_PRIMARY_TP=2
|
|
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
|
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
|
# engine default.
|
|
FRANK_PRIMARY_TP_WIRE=wht6
|
|
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
|
|
# start"); raise only after a real-traffic look at slot occupancy and the
|
|
# KV pool's headroom line. The recurrent-state cost scales with it (54
|
|
# MiB/copy/card, 2 copies with speculation).
|
|
FRANK_PRIMARY_MAX_SEQS=2
|
|
# Served window. Native is 262,144; 200,000 is the publisher's production
|
|
# value (compose) and what models.yml's contextWindow must quote — a longer
|
|
# prompt is refused by the engine, never truncated.
|
|
FRANK_PRIMARY_MAX_MODEL_LEN=200000
|
|
FRANK_PRIMARY_PLACEMENT=expert_tiered
|
|
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
|
|
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
|
|
# publisher shape's ~29 GiB — this pool is the cost of native precision.
|
|
# The boot ledger prints the pool's coverage ("the pool backs N% of the
|
|
# worst case ... M full-length sessions"); resize from that line, not from
|
|
# arithmetic.
|
|
FRANK_PRIMARY_HOST_POOL_MIB=40960
|
|
# VRAM left unclaimed per card (publisher default 96 MiB).
|
|
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
|
|
# Weights share of claimed VRAM (publisher production value; higher = more
|
|
# prefix-cache hit rate, less KV headroom).
|
|
FRANK_PRIMARY_EXPERT_VS_CACHE=0.82
|
|
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
|
|
# 2 KV heads, 256 head_dim, K+V).
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
|
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
|
# flag states it so the ledger records the intent).
|
|
FRANK_PRIMARY_SPEC_TOKENS=3
|
|
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
|
|
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
|
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
|
|
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
|
|
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
|
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
|
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
|
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
|
|
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
|
|
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
|
|
# plan, so the serve script refuses to start until both are raised. This is
|
|
# the floor it enforces (in MiB), with ~8 GiB of margin.
|
|
FRANK_PRIMARY_TTM_MIN_MIB=140000
|
|
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
|
# multi-flag values as one string; the serve script expands it unquoted on
|
|
# purpose.
|
|
FRANK_PRIMARY_EXTRA_ARGS=
|
|
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
|
# guaranteed long-timeout boot instead of a fast failure). The converted
|
|
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
|
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
|
# tail) - floor sits just under the payload.
|
|
FRANK_PRIMARY_RAD_MIN_MIB=177000
|