frank-default-large: split 0.68,0.32 = 45 blocks on 3090 (Q6_K phase 2)

Max for the Q6_K + q8_0/f16 KV config: 45 blocks (0-44), 11 FA layers
(3..43), KV 8624 MiB, ~0.7 GiB free. 46 layers -> ~0.4 GiB margin; 48
layers (f>=0.7197) adds FA block 47 -> KV 9408 -> OOM by ~0.85 GiB.
Draft model measured on ROCm0 only (no worker impact). Phase 1 (0.53)
verified: acceptance 0.833, KV 6272/6272, flash_attn auto, healthy.
This commit is contained in:
Frank
2026-08-17 02:05:04 -07:00
parent 1bc39c43ce
commit 11f37dafd8
+13 -7
View File
@@ -25,13 +25,19 @@ LISTEN_PORT=7770
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33). Phase-1 value
# for the Q6_K swap (2026-08-17); the split will be re-maximized after
# measuring the draft-model placement on the worker. With the Q6_K model
# (layer pool 20334 MB) + q8_0/f16 KV (784 MiB per FA layer) the worker
# frees ~4.7 GiB vs the old model at this split; the 0.53 boundary was
# provably safe under the old model and is comfortably safe here.
TENSOR_SPLIT=0.53,0.47
# 0.68,0.32 = 45 of 65 blocks on the 3090 (blocks 0-44) - the maximum
# for the Q6_K + q8_0/f16 KV config, measured 2026-08-17 phase 2:
# - KV per FA layer = 784 MiB (K 1088 B/token + V 2048 B/token @262144)
# - weights(f) = 20334 MB layer pool x f (GGUF scan, exact); embd on
# the head side (RPC0 buffer observed 10429.88 MiB at f=0.53)
# - KV-per-device count = round(f x 66) layers -> 45 layers = 11 FA
# (3..43) -> 8624 MiB. 46 layers (f>=0.7045) still 11 FA but margin
# drops to ~0.4 GiB; 48 layers (f>=0.7197) adds FA block 47 -> KV
# 9408 -> OOM by ~0.85 GiB.
# Predicted: 13430 (weights) + 8624 (KV) + ~530 (RS) + 189 (compute) =
# ~22773 MiB -> ~0.7 GiB free. Draft model runs on ROCm0 only (measured
# phase 1: 1718 MiB, no worker impact).
TENSOR_SPLIT=0.68,0.32
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).