frank-medium: orcarouter Q8_0 + DFlash2 Q4_K_M, 2x160k, balanced 32:33
Operator call 2026-09-19: switch the target to orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 (gated repo; the fetch script now requires an HF token and verifies sha256, not just size), take the drafter to the Q4_K_M rung of z-lab/Qwen3.8-27B-DFlash2-GGUF, cut the pool to two 163840-token conversations (327680, entirely inside the trained 262144 window), and balance the tensors across the cards (32:33) in place of the 35:30 x16-lean. The in-file MTP head stays one SPEC_TYPE change away.
This commit is contained in:
Frank committed
1 parent
374041cd7a
commit
d374ab963f
4 files changed
+210
-236
No files matched your search
+80
-139
@@ -1,7 +1,7 @@
|
||||
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
||||
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
||||
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
||||
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
||||
# Tunables for frank-medium (orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 +
|
||||
# z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M drafter), mainline llama.cpp on
|
||||
# Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this is
|
||||
# the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
||||
# routing proxy are retired.
|
||||
#
|
||||
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
||||
@@ -31,17 +31,17 @@
|
||||
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
||||
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
||||
#
|
||||
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
|
||||
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
|
||||
# own quant of the same size, on Unsloth's importance matrix). Abliteration is
|
||||
# orcarouter's single refusal direction projected out of 131 tensors; the vision
|
||||
# tower loads from mmproj-BF16.gguf (see NO_MMPROJ). The file also
|
||||
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
||||
# --spec-type draft-mtp is available without a second copy in VRAM.
|
||||
# Model: orcarouter/Qwen3.8-27B-Uncensored-GGUF @ Q8_0, 27.0 GiB, arch qwen35
|
||||
# (verified from the GGUF header 2026-09-19: 262144 trained window, 24 heads /
|
||||
# 4 KV heads, nextn_predict_layers=1). The file ALSO carries the model's MTP
|
||||
# head (blk.64.nextn.*), so --spec-type draft-mtp is available without a second
|
||||
# copy in VRAM -- one SPEC_TYPE change, the drafter below stays on disk.
|
||||
#
|
||||
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
||||
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q4_K_M, 1.06 GiB (operator call
|
||||
# 2026-09-19: the Q4 rung -- the drafter's job is throughput, not output
|
||||
# quality, and the target verifies every drafted token). 5 layers, hidden
|
||||
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
|
||||
# window. Decoding stays lossless -- the target verifies every drafted token.
|
||||
# window (dflash.attention.sliding_window=2048, verified from the header).
|
||||
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
|
||||
# (~40 MiB) instead of spanning the context (see SIZING).
|
||||
#
|
||||
@@ -52,10 +52,14 @@
|
||||
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
||||
# state at any depth.
|
||||
#
|
||||
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
||||
# the drafter AND its KV ride the second. The loader's compute buffers are part
|
||||
# of what is seen below -- they are allocated at load, not per request, and they
|
||||
# scale with --ubatch-size:
|
||||
# SIZING, the deployed shape (32:33 / ctx 327680 / 6144x512 / two slots /
|
||||
# DFlash2 Q4_K_M), measured 2026-09-19 after the model swap (rocm-smi):
|
||||
#
|
||||
# card 0 (x16) 26.9 GiB used, 5.7 GiB free
|
||||
# card 1 (x1) 27.8 GiB used, 4.8 GiB free
|
||||
#
|
||||
# The predecessor's measured table (rocm-smi, 2026-09-16, UD-Q8_K_XL target)
|
||||
# stays as the calibration history:
|
||||
#
|
||||
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
|
||||
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
|
||||
@@ -71,7 +75,7 @@
|
||||
# batch block below):
|
||||
#
|
||||
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
|
||||
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB <- deployed
|
||||
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB
|
||||
#
|
||||
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
|
||||
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
|
||||
@@ -80,36 +84,27 @@
|
||||
# were measured, which is why the balance is found by measurement rather than by
|
||||
# interpolating between splits.
|
||||
#
|
||||
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
||||
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
||||
# at 114688 was the vLLM predecessor's, not this runtime's. If a future change
|
||||
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
||||
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
||||
# WHY THE SPLIT IS 32:33 (operator call, 2026-09-19). The operator asked for
|
||||
# the tensors BALANCED across the two cards, which for 65 layers is 32:33 --
|
||||
# the most even split that exists. The predecessor's 35:30 skew existed to lean
|
||||
# work onto the x16 card (ROCm0), because ROCm1's Gen4 x1 chipset path made it
|
||||
# the 98 % utilization bottleneck; the measured trade of the balanced point is
|
||||
# that the x1 card runs hotter while the x16 card idles a little more, and the
|
||||
# operator accepted that for symmetry. Do not re-skew without asking.
|
||||
#
|
||||
# WHY THE SPLIT IS 35:30. Three effects decide it, and all three are measured:
|
||||
# * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's
|
||||
# weights, the draft graph and the last target layers, so it takes fewer
|
||||
# than half the layer count.
|
||||
# * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays
|
||||
# ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on
|
||||
# ROCm1 at this shape. Read the other way, that is exactly why the smaller
|
||||
# batch moved the balance toward ROCm1 -- and cost ROCm0 one layer.
|
||||
# * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE. One
|
||||
# -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the
|
||||
# reachable minimum gap is comparable to one step rather than to zero. 35:30
|
||||
# sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at
|
||||
# 36:29.
|
||||
# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5
|
||||
# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE
|
||||
# and the full window stays.
|
||||
# THE TWO CARDS' PATHS ARE NOT SYMMETRIC (measured, 2026-09-17): ROCm0
|
||||
# (03:00.0) hangs off CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s);
|
||||
# ROCm1 (0b:00.0) rides the B650 chipset through bridges that each negotiate
|
||||
# Gen4 x1 (~1.97 GB/s). The layer-split boundary itself is not the cost
|
||||
# (~5.2 MB of activation per 512-token ubatch rides the wire); the x1 card is
|
||||
# COMPUTE-bound, not PCIe-bound.
|
||||
#
|
||||
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
|
||||
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
|
||||
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
|
||||
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
|
||||
# has not reproduced (36:29 served this file's verification battery and the e2e
|
||||
# gate, and 35:30 is the deployed shape), but the signature is worth recognising
|
||||
# -- it is NOT an OOM, and no amount of headroom prevents it.
|
||||
# has not reproduced since, but the signature is worth recognising -- it is NOT
|
||||
# an OOM, and no amount of headroom prevents it.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
||||
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
||||
@@ -119,15 +114,15 @@
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
||||
|
||||
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
||||
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
||||
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
# The BF16 vision projector, same repo as the target. llama-server autoloads
|
||||
# any mmproj*.gguf beside the model; the serve script refuses to start without
|
||||
# it when vision is enabled, so a missing file is a deploy error, not a
|
||||
# text-only server that pretends to be multimodal.
|
||||
MM_PROJ_FILE=mmproj-BF16.gguf
|
||||
MODEL_DIR=/home/dank/models/Qwen3.8-27B-Uncensored-GGUF
|
||||
MODEL_FILE=Qwen3.8-27B-Uncensored-Q8_0.gguf
|
||||
# The f16 vision projector, same repo as the target. llama-server loads it via
|
||||
# the explicit --mmproj flag (no autoload in this build); the serve script
|
||||
# refuses to start without it when vision is enabled, so a missing file is a
|
||||
# deploy error, not a text-only server that pretends to be multimodal.
|
||||
MM_PROJ_FILE=mmproj-Qwen3.8-27B-Uncensored-f16.gguf
|
||||
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
||||
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
|
||||
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
||||
# at /v1/models and what every client asks for.
|
||||
@@ -141,64 +136,23 @@ HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
|
||||
# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card. The
|
||||
# two R9700s do NOT have symmetric PCIe paths. ROCm0 (03:00.0) hangs off
|
||||
# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
|
||||
# (0b:00.0) rides the B650 chipset through bridges that each negotiate
|
||||
# Gen4 x1 (~1.97 GB/s). Live rocm-smi showed the x1 card at 98% busy
|
||||
# against 40% on the x16 card: the MTP drafter, the output layers and the
|
||||
# heavier compute buffers are structurally pinned to the card holding the
|
||||
# LAST layers (the draft context shares the target's weights there), so
|
||||
# the bottlenecked card was doing most of the work. Moving one layer to
|
||||
# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
|
||||
# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
|
||||
# card the bigger share. The layer-split boundary itself is not the cost
|
||||
# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
|
||||
# card behind the x1 link should not be the saturated one while its
|
||||
# x16-sibling idles.
|
||||
#
|
||||
# WHY NOT FURTHER. One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
|
||||
# ~1.43 GiB (attention). The next step, 36:29, would cross the
|
||||
# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
|
||||
# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
|
||||
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
|
||||
# 0.58) is the farthest safe shift toward the healthy card at this pool.
|
||||
# Do not move it without re-measuring.
|
||||
|
||||
# MEASURED 2026-09-18 (isolated :7790, 2k-token decode, AVG rocm-smi util over
|
||||
# the request -- peak util lies, a layer split touches both cards every token):
|
||||
#
|
||||
# split draft_dev decode AVG util0(x16) AVG util1(x1)
|
||||
# 35,30 ROCm1 31.52 41 % 98 % <- deployed
|
||||
# 36,29 ROCm1 31.49 43 % 98 %
|
||||
# 35,30 ROCm0 31.52 42 % 98 %
|
||||
# 36,29 ROCm0 31.48 43 % 98 %
|
||||
#
|
||||
# The x1 card stays the 98 % bottleneck in EVERY configuration. The lever that
|
||||
# was tried is moving the MTP drafter to the x16 card (SPEC_DRAFT_DEVICE=ROCm0):
|
||||
# it does NOT flip the balance, because the drafter is a small nextn head and the
|
||||
# per-step cost is dominated by the VERIFY phase -- both cards run the full target
|
||||
# -- plus the LOGITS, which are pinned to the card holding the last layers. A
|
||||
# target-layer move (36:29) buys 2 points on the x16 card and erodes its VRAM
|
||||
# margin (fit: free 0.45 GiB) for nothing. The x1 card is COMPUTE-bound, not
|
||||
# PCIe-bound (~5.2 MB of activation per 512-token ubatch crosses the link);
|
||||
# saturating it costs no tokens beyond what any R9700 would. 35:30 is the
|
||||
# balance; do not chase the x16 card past it.
|
||||
TENSOR_SPLIT=35,30
|
||||
# 32:33 -- balanced (operator call 2026-09-19): 65 layers split as evenly as
|
||||
# they can be, ROCm1 (the last-layers card, which also carries the drafter and
|
||||
# the output layers) holding the extra one. See the WHY THE SPLIT IS 32:33
|
||||
# block above for the trade against the predecessor's 35:30 x16-lean.
|
||||
TENSOR_SPLIT=32,33
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
FIT=off
|
||||
FLASH_ATTN=on
|
||||
|
||||
# 393216 positions over TWO slots, so each conversation gets 196608 --
|
||||
# 327680 positions over TWO slots, so each conversation gets 163840 --
|
||||
# --parallel DIVIDES --ctx-size, so more slots would shrink every
|
||||
# conversation rather than grow the cache. The checkpoint's trained window is
|
||||
# 262144 and the GGUF carries NO rope scaling (freq_base 1e7, no YaRN keys),
|
||||
# so positions beyond 262144 extrapolate on the base rope: it runs, and 1.5x
|
||||
# is a modest ask of a rope, but the top 131k of the pool is untrained
|
||||
# territory.
|
||||
CTX_SIZE=393216
|
||||
# conversation rather than grow the cache. 163840 is inside the checkpoint's
|
||||
# trained 262144 window, so the whole pool is trained territory (no rope
|
||||
# extrapolation, unlike the predecessor's 393216).
|
||||
CTX_SIZE=327680
|
||||
PARALLEL=2
|
||||
|
||||
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
|
||||
@@ -258,20 +212,13 @@ CTX_CHECKPOINTS=0
|
||||
# (server_prompt_cache, tools/server/server-task.cpp): update() evicts
|
||||
# oldest-first past the cap, and alloc() REFUSES outright any single state
|
||||
# bigger than the whole cap. A state is the serialized KV of a whole sequence
|
||||
# plus its DeltaNet state plus the MTP draft context's own KV, so at this shape
|
||||
# it costs 68 KiB per token --
|
||||
# plus its DeltaNet state plus the draft context's own KV, so at this shape it
|
||||
# costs ~68 KiB per token --
|
||||
# 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB
|
||||
# plus 4096 B for the draft context (one nextn layer)
|
||||
# -- which makes the cap a TOKEN budget in disguise: 8192 MiB held ~120k
|
||||
# tokens, 16384 MiB holds ~240k.
|
||||
#
|
||||
# DOUBLED 8192 -> 16384 on 2026-09-18 because the old cap was BINDING, not
|
||||
# theoretical: alloc() refused 137 states in the two days from 2026-09-16 with
|
||||
# - prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping
|
||||
# i.e. 11.0-11.4 GiB states -- conversations of ~170k tokens at the law above --
|
||||
# so every deep conversation was silently uncacheable and re-prefilled from
|
||||
# scratch on its next turn. At 16384 a FULL 196608-token slot (~12.8 GiB) fits
|
||||
# with ~3 GiB of the budget spare for a second entry.
|
||||
# plus the draft context's window-bounded KV (a ~40 MiB constant for DFlash2)
|
||||
# -- which makes the cap a TOKEN budget in disguise: 16384 MiB holds ~240k
|
||||
# tokens, i.e. a FULL 163840-token slot (~10.6 GiB) with ~5 GiB of the budget
|
||||
# spare for a second entry.
|
||||
#
|
||||
# The cost is host RAM, allocated on demand and never pinned (plain heap
|
||||
# vectors), so no VRAM number above moves. dankserver has 62 GiB beside the
|
||||
@@ -300,24 +247,21 @@ REASONING_BUDGET=-1
|
||||
CHAT_TEMPLATE_FILE=
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
||||
|
||||
# Speculative decoding: the model's OWN MTP head -- the blk.<n>.nextn.*
|
||||
# tensors inside the target file. No sidecar file, no second copy of the
|
||||
# weights: the draft context runs on the main model (shares_model), so the
|
||||
# cost is one nextn layer's KV (4096 B/position) plus small buffers on the
|
||||
# card that holds the last layers. This is what makes the 393216 pool fit:
|
||||
# the DFlash2 sidecar's 1.92 GiB file and its buffers are off ROCm1.
|
||||
# n_max: 3 is the measured MTP winner (see the table below -- n=7 accepts 33 %
|
||||
# of 147, n=3 accepts 58 % of 72). The drafter rides ROCm1.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
# Speculative decoding: the DFlash2 sidecar (operator call 2026-09-19). The
|
||||
# target file also carries its own MTP head (nextn_predict_layers=1), so
|
||||
# draft-mtp is one SPEC_TYPE change away -- the drafter file stays on disk.
|
||||
# n_max 7 is the measured DFlash2 winner (see the table below); the drafter
|
||||
# rides ROCm1, the card that holds the last layers.
|
||||
SPEC_TYPE=draft-dflash
|
||||
SPEC_DRAFT_N_MAX=7
|
||||
SPEC_DRAFT_NGL=999
|
||||
SPEC_DRAFT_DEVICE=ROCm1
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
SPEC_DRAFT_TYPE_V=f16
|
||||
|
||||
# MTP (the model's own head) vs the DFlash2 drafter -- MEASURED 2026-09-16
|
||||
# (temperature 0, 1k prompt / 512 out; the autoregressive baseline and both
|
||||
# MTP depths were run in one session, same binary, same prompts):
|
||||
# DFlash2 vs the model's own MTP head -- MEASURED 2026-09-16 on the predecessor
|
||||
# payload (temperature 0, 1k prompt / 512 out; the autoregressive baseline and
|
||||
# both MTP depths were run in one session, same binary, same prompts):
|
||||
#
|
||||
# none 17.7 tok/s card1 26.07 GB
|
||||
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
|
||||
@@ -326,11 +270,10 @@ SPEC_DRAFT_TYPE_V=f16
|
||||
#
|
||||
# All four produced IDENTICAL greedy text, so verification is lossless
|
||||
# whichever proposer runs. DFlash2 is the faster proposer (2.46x over
|
||||
# autoregressive, 34 % over the best MTP); the operator switched to MTP on
|
||||
# 2026-09-17 for the VRAM, not the speed: the MTP head adds ~3.3 GB to the
|
||||
# card that holds it, against 2.06 GB for the drafter's file, and it is the
|
||||
# only drafter that fits the 393216 pool. Do not switch this without
|
||||
# re-measuring.
|
||||
# autoregressive, 34 % over the best MTP). The MTP head adds ~3.3 GB to the
|
||||
# card that holds it, against 1.06 GB for the Q4_K_M drafter's file -- at the
|
||||
# 327680 pool the drafter's window-bounded KV (~40 MiB) leaves headroom on
|
||||
# both cards, so the speed wins.
|
||||
#
|
||||
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
|
||||
# context type MTP requested but model doesn't contain MTP layers
|
||||
@@ -344,7 +287,7 @@ SPEC_DRAFT_TYPE_V=f16
|
||||
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
|
||||
# no model file, no draft context). It joins the chain unconditionally --
|
||||
# unlike MTP, which needs a draft context and therefore cannot share one with
|
||||
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-mtp alone, and the serve
|
||||
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-dflash alone, and the serve
|
||||
# script only passes --spec-ngram-mod-* when the type contains "ngram". The
|
||||
# window below is retained because the DFlash2 + ngram chain was measured free
|
||||
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
|
||||
@@ -352,7 +295,7 @@ SPEC_DRAFT_TYPE_V=f16
|
||||
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
|
||||
#
|
||||
# n_min/n_max prose 1k/512 prompt-echo 1.5k/400
|
||||
# 48/64 43.9 200.6 <- deployed
|
||||
# 48/64 43.9 200.6
|
||||
# 16/32 36.0 161.0
|
||||
# 8/16 36.2 113.8
|
||||
#
|
||||
@@ -369,9 +312,7 @@ SPEC_NGRAM_N_MATCH=24
|
||||
SPEC_NGRAM_N_MIN=48
|
||||
SPEC_NGRAM_N_MAX=64
|
||||
|
||||
# Vision enabled: llama-server autoloads mmproj-BF16.gguf from MODEL_DIR
|
||||
# (888 MiB, lands on ROCm0 -- the fit above charges it there). frank-small
|
||||
# (Dankclaw, RTX 3090) remains the fleet's dedicated image endpoint; this
|
||||
# makes frank-medium multimodal too, which is why the seam flips from
|
||||
# "text only" to "vision loads with the model".
|
||||
# Vision enabled: llama-server loads mmproj-Qwen3.8-27B-Uncensored-f16.gguf
|
||||
# (931 MiB, lands on ROCm0 -- the fit charges it there) via the explicit
|
||||
# --mmproj flag. This keeps frank-medium multimodal across the model swap.
|
||||
NO_MMPROJ=0
|
||||
Reference in new issue
Block a user