Frank
d317d6b246
Blocks 0-4 on ROCm0 + 45-47 on ROCm1. This exact shape was probed and battery-verified earlier the same session: card0 2.54 GiB free at boot, 2.11 GiB after the vision graph (steady under load), card1 unchanged at ~3.3 GiB. Card1 cannot take another block (0.82 GiB would OOM the vision graph + boot drift), so 5/3 is the physical limit at the 4x262144 / V-q8_0 shape. OT assertion pins the new value.
114 lines
6.1 KiB
Bash
114 lines
6.1 KiB
Bash
# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X)
|
||
# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO
|
||
# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision
|
||
# projector, and the draft-mtp + ngram-mod spec chain.
|
||
#
|
||
# These are the tunables read by scripts/serve-frank-primary.sh. The flag
|
||
# list is owned by the serve script; this file only sets fleet values.
|
||
#
|
||
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
||
# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address
|
||
# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets
|
||
# away with it; do not copy the shape.
|
||
|
||
# The payload (scripts/download-frank-primary.sh): the serve script refuses
|
||
# to start without all six Q8_0 parts, the MTP head and the projector.
|
||
MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
|
||
|
||
# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same
|
||
# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm
|
||
# runtime, no host SDK gate).
|
||
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
||
|
||
# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before
|
||
# the 2026-09-26 swap, so no client, probe or models.yml entry changes
|
||
# address.
|
||
BIND_HOST=0.0.0.0
|
||
PORT=7770
|
||
|
||
# Four conversations, each the model's full native 262144 window.
|
||
# --parallel divides --ctx-size into slots — the pool is 4x262144 cells
|
||
# (~23 GiB at the V-quantized law below, split across both cards). Slots
|
||
# DIVIDE the pool; they never multiply it. Per-slot window stays the
|
||
# full native 262144, so no client or models.yml change is implied.
|
||
CONTEXT_SIZE=1048576
|
||
PARALLEL_SLOTS=4
|
||
|
||
# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
|
||
# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
|
||
# optimum class for this pair at long contexts (the retired Dankclaw
|
||
# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
|
||
# acceptance); each extra verified position re-batches the CPU expert
|
||
# reads, so depth buys less than bandwidth here.
|
||
SPEC_DRAFT_N_MAX=2
|
||
|
||
# Device placement. -ngl 999 puts every layer's non-overridden tensors on
|
||
# the GPUs; the big overrides then move the bandwidth-huge tensors to host
|
||
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
||
# computing in RAM does not care). OT_OVERRIDES is ONE comma-separated
|
||
# <tensor-pattern>=<backend> list passed as a single --override-tensor
|
||
# flag (b11030 logs DEPRECATED for repeated flags and wants the joined
|
||
# form), and the loader applies the FIRST matching pattern per tensor —
|
||
# so the block-specific patterns MUST precede the broad exps=CPU sweep.
|
||
#
|
||
# Hot expert blocks stay GPU-resident (2026-09-26): -ot granularity is the
|
||
# whole per-block expert tensor (512 experts share one tensor), so "critical
|
||
# experts" = blocks co-located on the card that holds their layers.
|
||
# Measured cost: 2.42 GiB/block on ROCm0, 2.49 GiB/block on ROCm1.
|
||
# Blocks 0-4 on ROCm0 + 45-47 on ROCm1: card0 runs at ~2.5 GiB free at
|
||
# boot / ~2.1 GB under load (vision graph included) — the operator
|
||
# directed the fill PAST the 3 GB checkpoint floor on 2026-09-26
|
||
# ("shift a few more experts"); measured, battery-verified. Card1 keeps
|
||
# ~3.4 GB. Card1 CANNOT take another block: 3.31 GiB free − 2.49 GiB =
|
||
# 0.82 GiB, and the vision graph + boot drift alone exceed that.
|
||
# * exps=CPU — the remaining routed experts (blk.*.ffn_*_exps.weight,
|
||
# ~100 GiB at Q8_0 even with 7 blocks GPU-side), read once per token;
|
||
# the CPU backend computes them from page cache.
|
||
# * per_layer_token_embd.weight=CPU — the 51B-element PLE n-gram table
|
||
# (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
|
||
# sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
|
||
# the shared MTP head's borrow tensor — GPU-resident, the draft load
|
||
# crashes with "invalid vector subscript" (llama.cpp PR #28243) — so
|
||
# this override is REQUIRED, not just a placement choice. NEVER give
|
||
# it a GPU target (the 23 -> 13.4 t/s regression) and never --no-mmap
|
||
# (the table must stay page-cache-backed; anonymous memory
|
||
# cgroup-kills silently).
|
||
NGL=999
|
||
OT_OVERRIDES='blk\.[0-4]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU'
|
||
TENSOR_SPLIT=1,1
|
||
|
||
# The drafter MUST sit on the SECOND R9700 (ROCm1): with -ts 1,1 the main
|
||
# model's output head lands on the last-layers card, and the shared MTP head
|
||
# BORROWS it. --override-tensor sets has_tensor_overrides(), which pins
|
||
# n_copies==1 in the draft's scheduler — a borrowed output.weight sitting in
|
||
# a ROCm1 buffer then aborts the draft graph with ggml-backend.cpp:941
|
||
# "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run
|
||
# the operation" when the draft is elsewhere (verified live 2026-09-26: the
|
||
# ROCm0 placement crash-looped 4x at ~12 s, ROCm1 loaded and served).
|
||
SPEC_DRAFT_DEVICE=ROCm1
|
||
|
||
# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
|
||
# class on this fleet's ROCm stack (a bigger logical batch is free; compute
|
||
# buffers reserve at max graph size). ubatch 512 balances prefill
|
||
# throughput against the asymmetric compute-buffer cost on the card that
|
||
# holds the output layers and the drafter.
|
||
BATCH_SIZE=8192
|
||
UBATCH_SIZE=512
|
||
|
||
# KV cache types: V quantized to q8_0, K held at the f16 default. The V
|
||
# half is what quantizes on this build: main KV drops 24,576 -> 18,816
|
||
# B/token (12 of 48 layers carry KV; the 36 SSM layers are constant-state
|
||
# and never scale with ctx). NOTE the shared MTP drafter's KV ignores the
|
||
# V type (measured: draft pool 2,048 MiB = the f16 law at 1,048,576
|
||
# cells) — one layer's worth, so the miss costs ~0.5 GiB on ROCm1.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=q8_0
|
||
|
||
# Prompt-cache budget: 16 GiB. The V-quantized KV pool at four full
|
||
# windows is ~23 GiB spread across both cards (the hybrid DeltaNet/QSA
|
||
# pattern keeps even 4x262144 a fraction of a full-ctx x all-layers
|
||
# pool); the prompt cache is the RAM-side spend that actually moves
|
||
# prefill on repeat traffic. Host RAM budget: ~90 GiB of expert pages +
|
||
# this cache + the server's own arenas inside 246 GiB.
|
||
CACHE_RAM=16384
|