dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-small: two 128k slots, no spec, ubatch 512 (Frank's tight-fit order, 2026-10-06)

Browse Source 
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
This commit is contained in:
Frank
2026-10-06 00:07:03 -07:00
parent fb1a4fd150
commit 812d9ebff5
4 changed files with 113 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-small.env
+45 -36
View File
@@ -23,22 +23,29 @@
# => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes
# the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB).
#
# Why ONE slot at 131,072 is the maximum (Frank's cap per conversation):
# card 24,576 MiB. Weights 11,561 MiB (12,120,016,896 B, the IQ3_S-mtp
# file; the GSQ "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram
# IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ...
# all stock types, stock llama.cpp loads it). Projector ~1,350 MiB
# (927,607,488 B F16 file at the measured ~1.45x card law). Compute
# buffers at ubatch 1024 ~1.6 GiB. CUDA ctx ~500 MiB. DeltaNet state
# ~150 MiB/slot (48 linear layers x 48 value heads x 128x128 x 4 B).
# MTP draft context ~0.3-0.5 GiB (one nextn layer over the slot window;
# the boot ledger is the truth). Non-KV ~= 15.5 GiB. One q8_0 slot =
# 4,352 MiB => free ~= 4.5 GiB. A SECOND slot wants +4,352 MiB => free
# ~0.2 GiB, far below the 2-3 GiB floor. "As many 128k contexts as fit"
# = 1. MEASURE nvidia-smi free at boot (quote the free field): under
# ~1.5 GiB or a boot failure, step the pool down in 12,288-token
# increments (~410 MiB each at this rate). Compute buffers allocate at
# graph capture during BOOT, so a failure is loud and fast.
# 2026-10-06 (Frank): drop the MTP drafter (skip ngram too — on a qwen35
# hybrid, ngram speculation disables recurrent-state rollback: fleet lesson
# from the frank-large chain measurement), drop ubatch to 512, and take TWO
# conversations at 131,072 — "should be a very tight fit", accepted
# explicitly, i.e. the 2-3 GiB floor is knowingly violated. Measured at the
# 1x131072 MTP boot (2026-10-05): 19,404 used / 4,721 MiB free.
#
# Why TWO slots at 131,072 fits, by arithmetic (pool = 2 x 131,072 =
# 262,144; the per-conversation cap stays Frank's 128k):
#
# card 24,576 MiB. Weights 11,561 MiB (the IQ3_S-mtp file; the GSQ
# "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram IQ2_XS 144 /
# IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ... all stock
# types, stock llama.cpp loads it). Projector ~1,350 MiB (927,607,488 B
# F16 at the measured ~1.45x card law). Compute buffers at ubatch 512
# ~1.3 GiB (inferred from the measured 616 MiB/doubling rung on the 9B).
# CUDA ctx ~500 MiB. DeltaNet state ~150 MiB/slot x 2. NO draft context
# (the MTP draft ctx cost ~0.3-0.5 GiB and is gone with speculation).
# Non-KV ~15.5 GiB. Two q8_0 slots = 8,704 MiB (34 KiB/token x 131,072
# each). Predicted free at boot ~0.3-1.3 GiB. Compute buffers allocate
# at graph capture during BOOT, so a miss is loud and fast; MEASURE
# nvidia-smi free at boot (quote the free field) and let the runbook
# carry the number.
#
# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of
# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never
@@ -66,22 +73,23 @@ ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# ONE conversation at Frank's 128k cap: --ctx-size is the POOL llama.cpp
# divides across slots; 131072 / 1 = 131,072 per conversation. The cap is a
# product decision (Swift's own benchmark protocol ran at 131,072), and a
# second 131,072 slot does not fit the 2-3 GiB floor (arithmetic above).
CTX_SIZE=131072
PARALLEL=1
# TWO conversations at Frank's 128k cap (2026-10-06): --ctx-size is the
# POOL llama.cpp divides across slots; 262144 / 2 = 131,072 per
# conversation. Frank accepted the tight fit explicitly (arithmetic above);
# the per-slot window is the product contract, so models.yml stays at
# 131,072.
CTX_SIZE=262144
PARALLEL=2
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
# on this CUDA 3090 the curve climbs monotonically to 2048, but Frank's
# standing law for a resize is LOWER UBATCH FIRST (the measured 616 MiB rung
# on the 9B; the 27B's buffers are larger, so the claw-back is worth more).
# The pair is a measurement, not a law: if boot lands well above the floor,
# ubatch 2048 is the first thing to A/B for prefill (loud failure at graph
# capture if it does not fit).
# on the 9B the 1024->2048 rung measured +4.6 % prefill for 616 MiB, and
# Frank's 2026-10-06 order spends the other direction: ubatch 512 claws
# back ~0.3 GiB toward the second slot at some prefill cost. The pair is a
# measurement, not a law: ubatch 1024 is the first A/B back if a future
# boot lands with headroom (loud failure at graph capture if it does not
# fit).
BATCH_SIZE=4096
UBATCH_SIZE=1024
UBATCH_SIZE=512
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
@@ -92,12 +100,13 @@ UBATCH_SIZE=1024
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# Speculative decoding: the MTP head is IN the -mtp file (blk.64.nextn.*,
# nextn_predict_layers=1), so draft-mtp needs no second model and no second
# KV pool — the head shares the main model's weights. Depth 3 matches the
# fleet's radiance MTP depth; measure draft acceptance from the response
# timings object and retune here if the head earns more.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=3
# Speculative decoding is OFF (Frank, 2026-10-06): no draft-mtp, and
# ngram-mod is skipped too — on a qwen35 hybrid ngram disables
# recurrent-state rollback, so a rejected draft would corrupt the DeltaNet
# state. SPEC_TYPE=none strips BOTH speculative flags in the serve script
# (flags move as a set). The -mtp file still carries the head; it is simply
# not drafted with. Measured MTP reference (2026-10-05): 44.8 % acceptance
# at the recipe temperature => 58.9 t/s, 52.1 % at temp 0.8 => 66.1 t/s.
SPEC_TYPE=none
NGL=99
scripts/frank-small.service
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF (qwen35 hybrid, 1x131072, q8_0 K+V, in-file MTP head, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
Description=llama.cpp Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, no speculative decoding, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
After=network-online.target
Wants=network-online.target
scripts/serve-frank-small.sh
+31 -27
View File
@@ -3,11 +3,13 @@
# frank-small on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA
# sm_86).
#
# 2026-10-05 evening (Frank): a 27B on the 24 GiB card — ONE 131,072-token
# conversation (Frank's cap), KV at FP8 class (q8_0 both halves), the MTP
# head that lives IN the -mtp file drafting 3, vision on. The q8_0 KV
# contract supersedes the same-day f16/f16 rule, which was written for the
# Ornith 9B fit; the per-tensor flags stay explicit either way.
# 2026-10-06 (Frank): the 128k cap keeps the per-conversation window but
# the pool doubles to TWO 131,072 conversations (262144 / 2), speculative
# decoding is OFF (no draft-mtp; ngram skipped — on a qwen35 hybrid it
# disables recurrent-state rollback), and ubatch drops to 512. Frank
# accepted the tight fit explicitly (the 2-3 GiB floor is knowingly
# violated; frank-small.env carries the arithmetic). The q8_0 KV contract
# (FP8 class) is unchanged; the per-tensor flags stay explicit either way.
#
# Architecture: qwen35 hybrid — 64 real layers, 16 full-attention + 48
# Gated DeltaNet (full_attention_interval 4), native 262144 window, plus
@@ -40,29 +42,30 @@ fi
: "${ALIAS:=frank-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
# ctx-size is the POOL llama.cpp divides across slots: 131072 / 1 gives
# 131,072 per conversation — Frank's 128k cap. A second 131,072 slot does
# not fit the 2-3 GiB floor on the 24 GiB card with the 27B payload
# (frank-small.env carries the arithmetic); "as many 128k contexts as fit"
# = 1.
: "${CTX_SIZE:=131072}"
: "${PARALLEL:=1}"
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves),
# and Frank's standing law for a resize is LOWER UBATCH FIRST; the shipped
# pair is 4096/1024. If boot lands well above the 2-3 GiB floor, ubatch
# 2048 is the first A/B for prefill (a miss fails loud at graph capture).
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch.
# ctx-size is the POOL llama.cpp divides across slots: 262144 / 2 gives
# TWO 131,072 conversations — Frank's 128k cap each. 2026-10-06 (Frank):
# drop MTP, drop ubatch to 512, take two slots; the fit is tight by his
# explicit order (frank-small.env carries the arithmetic and the floor
# violation).
: "${CTX_SIZE:=262144}"
: "${PARALLEL:=2}"
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
# Frank's 2026-10-06 tight-fit order ships 4096/512 (claws back ~0.3 GiB
# toward the second slot); ubatch 1024 is the first A/B back if a future
# boot lands with headroom (a miss fails loud at graph capture). The
# logical batch is VRAM- and throughput-neutral at a fixed ubatch.
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=1024}"
: "${UBATCH_SIZE:=512}"
# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
# there is no true fp8 KV dtype). Explicit either way so the contract is
# visible in the argv.
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
# Speculative decoding: draft-mtp loads the MTP head that is IN the -mtp
# GGUF; the flags move as a set.
: "${SPEC_TYPE:=draft-mtp}"
: "${SPEC_DRAFT_N_MAX:=3}"
# Speculative decoding is OFF by default (Frank, 2026-10-06): SPEC_TYPE
# none strips BOTH speculative flags (flags move as a set). Setting a
# drafter here re-enables the pair — draft-mtp needs the head that is IN
# the -mtp GGUF.
: "${SPEC_TYPE:=none}"
: "${NGL:=99}"
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
@@ -80,11 +83,12 @@ if [[ -n "$MMPROJ_FILE" ]]; then
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
fi
# The MTP head ships IN the -mtp GGUF (blk.64.nextn.*,
# nextn_predict_layers=1): draft-mtp loads it against the main model's own
# weights — no --model-draft and no second KV pool. Speculative flags move
# as a set: a no-spec baseline must strip BOTH of these, not just one.
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
# Speculative flags move as a set: they reach the argv ONLY when a drafter
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
# baseline must not carry half a spec pair.
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
fi
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
# open after boot; make sure it exists before the server needs it.