diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 844451f..e944255 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -15,7 +15,7 @@ # (HIP runtime collision, see the 2026-08-13 TEMP incident). MODEL_DIR=/home/dank/models -MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf +MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_K.gguf HEAD_BIN=/home/dank/llama.cpp/build/bin/llama-server ALIAS=frank-default-large BIND_ADDR=0.0.0.0 @@ -25,15 +25,12 @@ LISTEN_PORT=7770 RPC_SERVERS=192.168.0.235:50052 # Model split fractions, one per device in order [RPC0, iGPU]. -# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33) - the maximum -# provably-safe split, measured 2026-08-17 (two failed attempts): -# - 0.50: 32 blocks, KV 8192 MiB (8 FA) -> healthy, 1477 MiB free. -# - 0.54: weights 35 blocks but KV jumps to 9216 MiB (9th FA block) -> -# cudaMalloc OOM by ~740 MiB. The KV-per-device count rounds the -# split fraction over 66 layers, independent of the weight boundary; -# the flip to 9 FA sits at f ~= 0.538, just below the weight -# threshold for block 35. 0.53 keeps 8 FA under every rounding rule. -# GGUF block scan: FA at 3,7,...,63; linear blocks 431 MB, FA 410 MB. +# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33). Phase-1 value +# for the Q6_K swap (2026-08-17); the split will be re-maximized after +# measuring the draft-model placement on the worker. With the Q6_K model +# (layer pool 20334 MB) + q8_0/f16 KV (784 MiB per FA layer) the worker +# frees ~4.7 GiB vs the old model at this split; the 0.53 boundary was +# provably safe under the old model and is comfortably safe here. TENSOR_SPLIT=0.53,0.47 # Context pool. 262144 total = 1 parallel slot x 262144 per slot @@ -41,25 +38,26 @@ TENSOR_SPLIT=0.53,0.47 CTX_SIZE=262144 PARALLEL=1 -# Speculative decoding. Matches production frank-default-large: MTP (bundled -# head, n_max 4 per the published tuning curve) with the ngram-mod fallback. -# Set SPEC_NONE=1 for a raw baseline with no speculation. +# Speculative decoding. MTP with the STANDALONE draft head (the Q6_K +# model has no bundled MTP head; the repo ships the draft sidecar) + the +# ngram-mod fallback. Set SPEC_NONE=1 for a raw baseline. SPEC_NONE=0 SPEC_DEFAULT=1 MTP=1 SPEC_DRAFT_N_MAX=4 -# Optional standalone MTP draft head (e.g. the Q4_0 ggml-org file). When set, -# it replaces the bundled in-model head via --spec-draft-model. -SPEC_DRAFT_MODEL= +# Standalone MTP draft head (Q8_0 MTP-only sidecar from the same repo). +SPEC_DRAFT_MODEL=/home/dank/models/Qwen3.8-27B-Uncensored-draft-Q8_0.gguf SPEC_NGRAM_MOD_N_MIN=48 SPEC_NGRAM_MOD_N_MAX=64 SPEC_NGRAM_MOD_N_MATCH=24 -# KV on GPU, unquantized (f16). 16 full-attention layers x 2 (K,V) x 4 heads -# x 256 dim x 2 bytes = 64 KiB/token -> 262144 tokens = 16.6 GiB total, -# held in proportion to the full-attention layers each device holds. -# Keep K == V so Flash Attention stays enabled. -CACHE_TYPE_K=f16 +# KV on GPU: K q8_0 + V f16 (mismatched, 16 full-attention layers x 4 +# heads x 256 dim: K = 1088 B/token, V = 2048 B/token -> 3136 B/token -> +# 784 MiB per FA layer at 262144, 12.25 GiB total). Mismatched K/V with +# Flash Attention requires the GGML_CUDA_FA_ALL_QUANTS rebuild (2026-08-17: +# head HIP + worker CUDA at 9b05354ec). Do not revert the binaries without +# also reverting this. +CACHE_TYPE_K=q8_0 CACHE_TYPE_V=f16 # Batch sizing: production-verified values (2026-08-14).