frank-primary: four full-window slots, V q8_0 KV, hot expert blocks on-GPU
- CONTEXT_SIZE=1048576 / PARALLEL_SLOTS=4: four conversations, each the full native 262144 window (per-slot window unchanged; slots divide the pool, they never multiply it). - CACHE_TYPE_V=q8_0 / CACHE_TYPE_K=f16: main KV drops 24,576 -> 18,816 B/token (-23.4%); the shared MTP drafter's KV ignores the V type (measured: 2,048 MiB draft pool = the f16 law at 1,048,576 cells). - OT_OVERRIDES: blocks 0-3 on ROCm0 and 45-47 on ROCm1 stay GPU-resident (2.42/2.49 GiB per block, measured); the rest stay CPU-computed. Measured free under load: ~4.9 GB card0 / ~3.4 GB card1; a 5th block on card0 lands at 2.27 GB, under the 3 GB checkpoint floor, so blocks are atomic and 4 is the safe fill. - serve script: --cache-type-k/v env-wired tunables; ONE joined --override-tensor flag (b11030 deprecates repeated flags, and the loader is first-match-wins, so the specific patterns must precede the broad exps=CPU sweep). - seams: the window test asserts per-slot = 262144 at 4 slots; new KV-type test; OT exact-value assertion pins the ordering law.
This commit is contained in:
Frank
+43
-20
@@ -26,30 +26,43 @@ RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
|
||||
BIND_HOST=0.0.0.0
|
||||
PORT=7770
|
||||
|
||||
# One conversation, the model's full native 262144 window.
|
||||
CONTEXT_SIZE=262144
|
||||
PARALLEL_SLOTS=1
|
||||
# Four conversations, each the model's full native 262144 window.
|
||||
# --parallel divides --ctx-size into slots — the pool is 4x262144 cells
|
||||
# (~23 GiB at the V-quantized law below, split across both cards). Slots
|
||||
# DIVIDE the pool; they never multiply it. Per-slot window stays the
|
||||
# full native 262144, so no client or models.yml change is implied.
|
||||
CONTEXT_SIZE=1048576
|
||||
PARALLEL_SLOTS=4
|
||||
|
||||
# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
|
||||
# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
|
||||
# optimum class for this pair at long contexts (the retired Dankclaw
|
||||
# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
|
||||
# acceptance); each extra verified position re-batches the CPU expert reads,
|
||||
# so depth buys less than bandwidth here.
|
||||
# acceptance); each extra verified position re-batches the CPU expert
|
||||
# reads, so depth buys less than bandwidth here.
|
||||
SPEC_DRAFT_N_MAX=2
|
||||
|
||||
# Device placement. -ngl 999 puts every layer's non-overridden tensors on
|
||||
# the GPUs; the big overrides then move the bandwidth-huge tensors to host
|
||||
# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
|
||||
# computing in RAM does not). OT_OVERRIDES is comma-separated
|
||||
# <tensor-pattern>=<backend> pairs passed as repeated --override-tensor
|
||||
# flags. Both overrides are pinned by the header dump of parts 2-3
|
||||
# (2026-09-26, /home/dank/models/dump-gguf.py — GGUF v3, 512 experts
|
||||
# top-10, qwen4exp.ple.ngram_size=3, heads_per_ngram=8):
|
||||
# * exps=CPU — the routed experts (blk.*.ffn_*_exps.weight, ~100 GiB at
|
||||
# Q8_0), read once per token; the CPU backend computes them from page
|
||||
# cache.
|
||||
# * per_layer_token_embd.weight=CPU — the 51.2B-element PLE n-gram table
|
||||
# computing in RAM does not care). OT_OVERRIDES is ONE comma-separated
|
||||
# <tensor-pattern>=<backend> list passed as a single --override-tensor
|
||||
# flag (b11030 logs DEPRECATED for repeated flags and wants the joined
|
||||
# form), and the loader applies the FIRST matching pattern per tensor —
|
||||
# so the block-specific patterns MUST precede the broad exps=CPU sweep.
|
||||
#
|
||||
# Hot expert blocks stay GPU-resident (2026-09-26): -ot granularity is the
|
||||
# whole per-block expert tensor (512 experts share one tensor), so "critical
|
||||
# experts" = blocks co-located on the card that holds their layers.
|
||||
# Measured cost: 2.42 GiB/block on ROCm0, 2.49 GiB/block on ROCm1.
|
||||
# Blocks 0-3 on ROCm0 + 45-47 on ROCm1 leave ~4.9 GB / ~3.4 GB free under
|
||||
# load (vision graph included). A 5th block on ROCm0 lands at 2.27 GB —
|
||||
# under the 3 GB checkpoint floor — so it stays off; blocks are atomic,
|
||||
# nothing lands card0 inside the 3-4 GB band.
|
||||
# * exps=CPU — the remaining routed experts (blk.*.ffn_*_exps.weight,
|
||||
# ~100 GiB at Q8_0 even with 7 blocks GPU-side), read once per token;
|
||||
# the CPU backend computes them from page cache.
|
||||
# * per_layer_token_embd.weight=CPU — the 51B-element PLE n-gram table
|
||||
# (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
|
||||
# sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
|
||||
# the shared MTP head's borrow tensor — GPU-resident, the draft load
|
||||
@@ -59,7 +72,7 @@ SPEC_DRAFT_N_MAX=2
|
||||
# (the table must stay page-cache-backed; anonymous memory
|
||||
# cgroup-kills silently).
|
||||
NGL=999
|
||||
OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU
|
||||
OT_OVERRIDES='blk\.[0-3]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU'
|
||||
TENSOR_SPLIT=1,1
|
||||
|
||||
# The drafter MUST sit on the SECOND R9700 (ROCm1): with -ts 1,1 the main
|
||||
@@ -80,9 +93,19 @@ SPEC_DRAFT_DEVICE=ROCm1
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=512
|
||||
|
||||
# Prompt-cache budget: 16 GiB. The KV itself is small (the hybrid
|
||||
# DeltaNet/QSA pattern keeps a 262144 window a fraction of the full-ctx
|
||||
# x all-layers number), so the prompt cache is the RAM-side spend that
|
||||
# actually moves prefill on repeat traffic. Host RAM budget: ~175 GiB of
|
||||
# expert pages + this cache + the server's own arenas inside 246 GiB.
|
||||
# KV cache types: V quantized to q8_0, K held at the f16 default. The V
|
||||
# half is what quantizes on this build: main KV drops 24,576 -> 18,816
|
||||
# B/token (12 of 48 layers carry KV; the 36 SSM layers are constant-state
|
||||
# and never scale with ctx). NOTE the shared MTP drafter's KV ignores the
|
||||
# V type (measured: draft pool 2,048 MiB = the f16 law at 1,048,576
|
||||
# cells) — one layer's worth, so the miss costs ~0.5 GiB on ROCm1.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Prompt-cache budget: 16 GiB. The V-quantized KV pool at four full
|
||||
# windows is ~23 GiB spread across both cards (the hybrid DeltaNet/QSA
|
||||
# pattern keeps even 4x262144 a fraction of a full-ctx x all-layers
|
||||
# pool); the prompt cache is the RAM-side spend that actually moves
|
||||
# prefill on repeat traffic. Host RAM budget: ~90 GiB of expert pages +
|
||||
# this cache + the server's own arenas inside 246 GiB.
|
||||
CACHE_RAM=16384
|
||||
|
||||
@@ -55,7 +55,8 @@ fi
|
||||
: "${SPEC_DRAFT_DEVICE:=ROCm0}"
|
||||
: "${BATCH_SIZE:=8192}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${CACHE_RAM:=16384}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${VERBOSITY:=4}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
||||
@@ -108,7 +109,8 @@ ARGS=(
|
||||
--reasoning auto
|
||||
--reasoning-format deepseek
|
||||
--reasoning-budget -1
|
||||
--flash-attn on
|
||||
--cache-type-k "$CACHE_TYPE_K"
|
||||
--cache-type-v "$CACHE_TYPE_V"
|
||||
--batch-size "$BATCH_SIZE"
|
||||
--ubatch-size "$UBATCH_SIZE"
|
||||
--cache-ram "$CACHE_RAM"
|
||||
@@ -128,10 +130,11 @@ ARGS=(
|
||||
--min-p 0.0
|
||||
)
|
||||
|
||||
# Comma-separated <pattern>=<backend> pairs -> repeated --override-tensor.
|
||||
IFS=',' read -ra _ots <<< "$OT_OVERRIDES"
|
||||
for ot in "${_ots[@]}"; do
|
||||
ARGS+=(--override-tensor "$ot")
|
||||
done
|
||||
# Comma-separated <pattern>=<backend> pairs, ONE flag: b11030 logs
|
||||
# DEPRECATED for repeated --override-tensor and wants the joined list.
|
||||
# The loader applies the FIRST matching pattern per tensor, so the
|
||||
# specific block patterns must precede the broad exps=CPU sweep in
|
||||
# frank-primary.env.
|
||||
ARGS+=(--override-tensor "$OT_OVERRIDES")
|
||||
|
||||
exec "$SERVER_BIN" "${ARGS[@]}"
|
||||
|
||||
Reference in New Issue
Block a user