Frank
a9b94a79eb
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts (scripts/dump-gguf.py, added — a dependency-free reader that never touches tensor data, so a 50 GiB part scans in milliseconds) before the first load. What the dump says (parts 2-3, 2026-09-26): - The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd. weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of part 3. It is the same tensor the shared MTP head borrows, so the =CPU override is required for placement AND for the draft load (GPU-resident borrow: "invalid vector subscript"). - The model is qwen4exp, 512 experts, top-10 routing (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). - Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors. The placeholder pattern ngram_embeddings matched nothing — dropped from OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook, LLM index) before it could masquerade as a working override at load time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU. Suite: 346 green.
83 lines
4.0 KiB
Bash
83 lines
4.0 KiB
Bash
# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X)
|
|
# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO
|
|
# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision
|
|
# projector, and the draft-mtp + ngram-mod spec chain.
|
|
#
|
|
# These are the tunables read by scripts/serve-frank-primary.sh. The flag
|
|
# list is owned by the serve script; this file only sets fleet values.
|
|
#
|
|
# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address
|
|
# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets
|
|
# away with it; do not copy the shape.
|
|
|
|
# The payload (scripts/download-frank-primary.sh): the serve script refuses
|
|
# to start without all six Q8_0 parts, the MTP head and the projector.
|
|
MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
|
|
|
|
# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same
|
|
# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm
|
|
# runtime, no host SDK gate).
|
|
RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
|
|
|
# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before
|
|
# the 2026-09-26 swap, so no client, probe or models.yml entry changes
|
|
# address.
|
|
BIND_HOST=0.0.0.0
|
|
PORT=7770
|
|
|
|
# One conversation, the model's full native 262144 window.
|
|
CONTEXT_SIZE=262144
|
|
PARALLEL_SLOTS=1
|
|
|
|
# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
|
|
# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
|
|
# optimum class for this pair at long contexts (the retired Dankclaw
|
|
# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
|
|
# acceptance); each extra verified position re-batches the CPU expert reads,
|
|
# so depth buys less than bandwidth here.
|
|
SPEC_DRAFT_N_MAX=2
|
|
|
|
# Device placement. -ngl 999 puts every layer's non-overridden tensors on
|
|
# the GPUs; the big overrides then move the bandwidth-huge tensors to host
|
|
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
|
# computing in RAM does not). OT_OVERRIDES is comma-separated
|
|
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
|
|
# flags. Both overrides are pinned by the header dump of parts 2-3
|
|
# (2026-09-26, /home/dank/models/dump-gguf.py — GGUF v3, 512 experts
|
|
# top-10, qwen4exp.ple.ngram_size=3, heads_per_ngram=8):
|
|
# * exps=CPU — the routed experts (blk.*.ffn_*_exps.weight, ~100 GiB at
|
|
# Q8_0), read once per token; the CPU backend computes them from page
|
|
# cache.
|
|
# * per_layer_token_embd.weight=CPU — the 51.2B-element PLE n-gram table
|
|
# (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
|
|
# sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
|
|
# the shared MTP head's borrow tensor — GPU-resident, the draft load
|
|
# crashes with "invalid vector subscript" (llama.cpp PR #28243) — so
|
|
# this override is REQUIRED, not just a placement choice. NEVER give
|
|
# it a GPU target (the 23 -> 13.4 t/s regression) and never --no-mmap
|
|
# (the table must stay page-cache-backed; anonymous memory
|
|
# cgroup-kills silently).
|
|
NGL=999
|
|
OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU
|
|
TENSOR_SPLIT=1,1
|
|
|
|
# The drafter lives on the first R9700 (the card the load log shows holding
|
|
# the shared embedding borrow path).
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
|
|
# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
|
|
# class on this fleet's ROCm stack (a bigger logical batch is free; compute
|
|
# buffers reserve at max graph size). ubatch 512 balances prefill
|
|
# throughput against the asymmetric compute-buffer cost on the card that
|
|
# holds the output layers and the drafter.
|
|
BATCH_SIZE=8192
|
|
UBATCH_SIZE=512
|
|
|
|
# Prompt-cache budget: 16 GiB. The KV itself is small (the hybrid
|
|
# DeltaNet/QSA pattern keeps a 262144 window a fraction of the full-ctx
|
|
# x all-layers number), so the prompt cache is the RAM-side spend that
|
|
# actually moves prefill on repeat traffic. Host RAM budget: ~175 GiB of
|
|
# expert pages + this cache + the server's own arenas inside 246 GiB.
|
|
CACHE_RAM=16384
|