dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: MTP drafter, 393216 pool (2x196k), split 34:31

Browse Source 
DFlash2's sidecar (1.92 GiB + buffers on ROCm1) kept the 393216 pool off the table; the MTP head lives in the target file, so its cost is one nextn layer's KV (4096 B/pos) plus small buffers. Fit: ROCm0 29.77 / ROCm1 28.32 GiB, over the floor, 34:31 the balance point at this pool. n_max 3 is the measured MTP winner. The seam's fit is now drafter-aware, the slack is the deployed integer gap (1483 MiB), the ctx cap 393216 with the rope caveat (top 131k extrapolates on the base rope).
This commit is contained in:
Frank
2026-09-17 02:56:36 -07:00
parent dbd764a493
commit 23f09788e9
2 changed files with 92 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-medium.env
+35 -36
View File
@@ -141,30 +141,26 @@ HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 36:29 -- one layer over the 35:30 balance, moved to buy headroom on ROCm1,
# the card that carries the drafter, its KV, the output layers and the heavier
# compute buffers, now that the context grew to 327680 (see below). Fit at
# the deployed point (36:29, ctx 327680, batch 6144 x ubatch 512, two slots,
# DFlash2 + ngram-mod, mmproj on ROCm0): ROCm0 29.19 / ROCm1 26.04 GiB of
# 31.86, i.e. 2.67 / 5.82 GiB free and a 3.15 GiB gap BY CONSTRUCTION -- the
# balance point at this context is 34:31 (0.11 GiB gap) and the operator chose
# headroom on the drafter card over balance. See SIZING above for the
# measurement table; one -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or
# ~1.43 GiB (attention), so do not move it without re-measuring.
TENSOR_SPLIT=36,29
# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step
# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
# move it without re-measuring.
TENSOR_SPLIT=34,31
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
FIT=off
FLASH_ATTN=on
# 327680 positions over TWO slots, so each conversation gets 163840 --
# 393216 positions over TWO slots, so each conversation gets 196608 --
# --parallel DIVIDES --ctx-size, so more slots would shrink every
# conversation rather than grow the cache. The checkpoint's trained window is
# 262144 and the GGUF carries NO rope scaling (freq_base 1e7, no YaRN keys),
# so positions beyond 262144 extrapolate on the base rope: it runs, and 1.25x
# is a modest ask of it, but the extra 65k is untrained territory.
CTX_SIZE=327680
# so positions beyond 262144 extrapolate on the base rope: it runs, and 1.5x
# is a modest ask of a rope, but the top 131k of the pool is untrained
# territory.
CTX_SIZE=393216
PARALLEL=2
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
@@ -238,34 +234,36 @@ REASONING_BUDGET=-1
CHAT_TEMPLATE_FILE=
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
# Speculative decoding: the DFlash2 block-diffusion drafter PLUS the draftless
# n-gram matcher. DFlash2's GGUF carries block_size 8, so at most 7 draft
# tokens per verification step (llama.cpp asserts n_max <= block_size - 1).
# The drafter rides ROCm1, the card that ends up holding the target's last
# layers.
SPEC_TYPE=draft-dflash,ngram-mod
SPEC_DRAFT_N_MAX=7
# Speculative decoding: the model's OWN MTP head -- the blk.<n>.nextn.*
# tensors inside the target file. No sidecar file, no second copy of the
# weights: the draft context runs on the main model (shares_model), so the
# cost is one nextn layer's KV (4096 B/position) plus small buffers on the
# card that holds the last layers. This is what makes the 393216 pool fit:
# the DFlash2 sidecar's 1.92 GiB file and its buffers are off ROCm1.
# n_max: 3 is the measured MTP winner (see the table below -- n=7 accepts 33 %
# of 147, n=3 accepts 58 % of 72). The drafter rides ROCm1.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=3
SPEC_DRAFT_NGL=999
SPEC_DRAFT_DEVICE=ROCm1
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16
# MTP (the model's own head) vs the drafter -- MEASURED 2026-09-16 (temperature
# 0, 1k prompt / 512 out; the autoregressive baseline and both MTP depths were
# run against the deployed configuration in one session, same binary, same
# prompts):
# MTP (the model's own head) vs the DFlash2 drafter -- MEASURED 2026-09-16
# (temperature 0, 1k prompt / 512 out; the autoregressive baseline and both
# MTP depths were run in one session, same binary, same prompts):
#
# none 17.7 tok/s card1 26.07 GB
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
#
# All four produced IDENTICAL greedy text, so verification is lossless whichever
# proposer runs. DFlash2 is worth 2.46x over autoregressive and 34 % over the
# best MTP; deeper MTP drafting costs more than it returns (n=7 accepts 33 % of
# 147 proposals, n=3 accepts 58 % of 72). The MTP head is not free either:
# selecting it loads blk.64.nextn.* and adds ~3.3 GB to the card that holds it,
# against 2.06 GB for the drafter's file. Do not switch this without
# All four produced IDENTICAL greedy text, so verification is lossless
# whichever proposer runs. DFlash2 is the faster proposer (2.46x over
# autoregressive, 34 % over the best MTP); the operator switched to MTP on
# 2026-09-17 for the VRAM, not the speed: the MTP head adds ~3.3 GB to the
# card that holds it, against 2.06 GB for the drafter's file, and it is the
# only drafter that fits the 393216 pool. Do not switch this without
# re-measuring.
#
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
@@ -278,11 +276,12 @@ SPEC_DRAFT_TYPE_V=f16
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
# no model file, no draft context). It joins the chain unconditionally --
# no model file, no draft context). It joins the chain unconditionally --
# unlike MTP, which needs a draft context and therefore cannot share one with
# DFlash2. Verified on the loaded server: with ngram-mod as the ONLY spec type,
# a prose prompt produces ZERO drafts (plain autoregressive) while a
# prompt-echo prompt accepts 100 % of them, so its drafts are unmistakable.
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-mtp alone, and the serve
# script only passes --spec-ngram-mod-* when the type contains "ngram". The
# window below is retained because the DFlash2 + ngram chain was measured free
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
#
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
#