frank-default-large: split 0.68,0.32 = 45 blocks on 3090 (Q6_K phase 2)
Max for the Q6_K + q8_0/f16 KV config: 45 blocks (0-44), 11 FA layers (3..43), KV 8624 MiB, ~0.7 GiB free. 46 layers -> ~0.4 GiB margin; 48 layers (f>=0.7197) adds FA block 47 -> KV 9408 -> OOM by ~0.85 GiB. Draft model measured on ROCm0 only (no worker impact). Phase 1 (0.53) verified: acceptance 0.833, KV 6272/6272, flash_attn auto, healthy.
This commit is contained in:
@@ -25,13 +25,19 @@ LISTEN_PORT=7770
|
||||
RPC_SERVERS=192.168.0.235:50052
|
||||
|
||||
# Model split fractions, one per device in order [RPC0, iGPU].
|
||||
# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33). Phase-1 value
|
||||
# for the Q6_K swap (2026-08-17); the split will be re-maximized after
|
||||
# measuring the draft-model placement on the worker. With the Q6_K model
|
||||
# (layer pool 20334 MB) + q8_0/f16 KV (784 MiB per FA layer) the worker
|
||||
# frees ~4.7 GiB vs the old model at this split; the 0.53 boundary was
|
||||
# provably safe under the old model and is comfortably safe here.
|
||||
TENSOR_SPLIT=0.53,0.47
|
||||
# 0.68,0.32 = 45 of 65 blocks on the 3090 (blocks 0-44) - the maximum
|
||||
# for the Q6_K + q8_0/f16 KV config, measured 2026-08-17 phase 2:
|
||||
# - KV per FA layer = 784 MiB (K 1088 B/token + V 2048 B/token @262144)
|
||||
# - weights(f) = 20334 MB layer pool x f (GGUF scan, exact); embd on
|
||||
# the head side (RPC0 buffer observed 10429.88 MiB at f=0.53)
|
||||
# - KV-per-device count = round(f x 66) layers -> 45 layers = 11 FA
|
||||
# (3..43) -> 8624 MiB. 46 layers (f>=0.7045) still 11 FA but margin
|
||||
# drops to ~0.4 GiB; 48 layers (f>=0.7197) adds FA block 47 -> KV
|
||||
# 9408 -> OOM by ~0.85 GiB.
|
||||
# Predicted: 13430 (weights) + 8624 (KV) + ~530 (RS) + 189 (compute) =
|
||||
# ~22773 MiB -> ~0.7 GiB free. Draft model runs on ROCm0 only (measured
|
||||
# phase 1: 1718 MiB, no worker impact).
|
||||
TENSOR_SPLIT=0.68,0.32
|
||||
|
||||
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
|
||||
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).
|
||||
|
||||
Reference in New Issue
Block a user