# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X)
# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO
# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision
# projector, and the draft-mtp + ngram-mod spec chain.
#
# These are the tunables read by scripts/serve-frank-primary.sh. The flag
# list is owned by the serve script; this file only sets fleet values.
#
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address
# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets
# away with it; do not copy the shape.

# The payload (scripts/download-frank-primary.sh): the serve script refuses
# to start without all six Q8_0 parts, the MTP head and the projector.
MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF

# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same
# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm
# runtime, no host SDK gate).
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030

# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before
# the 2026-09-26 swap, so no client, probe or models.yml entry changes
# address.
BIND_HOST=0.0.0.0
PORT=7770

# One conversation, the model's full native 262144 window.
CONTEXT_SIZE=262144
PARALLEL_SLOTS=1

# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
# optimum class for this pair at long contexts (the retired Dankclaw
# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
# acceptance); each extra verified position re-batches the CPU expert reads,
# so depth buys less than bandwidth here.
SPEC_DRAFT_N_MAX=2

# Device placement. -ngl 999 puts every layer's non-overridden tensors on
# the GPUs; the big overrides then move the bandwidth-huge tensors to host
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
# computing in RAM does not). OT_OVERRIDES is comma-separated
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
# flags:
#   * exps=CPU — the routed experts. ~100 GiB at Q8, read once per token;
#     the CPU backend computes them from page cache.
#   * the n-gram embedding block (the 51B-param PLE table) — sparse row
#     lookups, tiny PCIe traffic from CPU RAM; NEVER pin with a GPU target
#     (the 23 -> 13.4 t/s regression).
#   * per_layer_token_embd.weight=CPU — REQUIRED by the shared MTP head's
#     borrow path: GPU-resident, it crashes the draft load with
#     "invalid vector subscript" (llama.cpp PR #28243).
# The exact tensor pattern for the n-gram block was discovered at load time
# on 2026-09-26 (see the runbook); this list is the deployed set.
NGL=999
OT_OVERRIDES=exps=CPU,ngram_embeddings=CPU,per_layer_token_embd.weight=CPU
TENSOR_SPLIT=1,1

# The drafter lives on the first R9700 (the card the load log shows holding
# the shared embedding borrow path).
SPEC_DRAFT_DEVICE=ROCm0

# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
# class on this fleet's ROCm stack (a bigger logical batch is free; compute
# buffers reserve at max graph size). ubatch 512 balances prefill
# throughput against the asymmetric compute-buffer cost on the card that
# holds the output layers and the drafter.
BATCH_SIZE=8192
UBATCH_SIZE=512

# Prompt-cache budget: 16 GiB. The KV itself is small (the hybrid
# DeltaNet/QSA pattern keeps a 262144 window a fraction of the full-ctx
# x all-layers number), so the prompt cache is the RAM-side spend that
# actually moves prefill on repeat traffic. Host RAM budget: ~175 GiB of
# expert pages + this cache + the server's own arenas inside 246 GiB.
CACHE_RAM=16384