frank-default-large: split 0.68,0.32 = 45 blocks on 3090 (Q6_K phase 2)
Max for the Q6_K + q8_0/f16 KV config: 45 blocks (0-44), 11 FA layers (3..43), KV 8624 MiB, ~0.7 GiB free. 46 layers -> ~0.4 GiB margin; 48 layers (f>=0.7197) adds FA block 47 -> KV 9408 -> OOM by ~0.85 GiB. Draft model measured on ROCm0 only (no worker impact). Phase 1 (0.53) verified: acceptance 0.833, KV 6272/6272, flash_attn auto, healthy.
This commit is contained in:
1 parent
1bc39c43ce
commit
11f37dafd8
1 file changed
+13
-7
@@ -25,13 +25,19 @@ LISTEN_PORT=7770
|
|||||||
RPC_SERVERS=192.168.0.235:50052
|
RPC_SERVERS=192.168.0.235:50052
|
||||||
|
|
||||||
# Model split fractions, one per device in order [RPC0, iGPU].
|
# Model split fractions, one per device in order [RPC0, iGPU].
|
||||||
# 0.53,0.47 = 34 of 65 blocks on the 3090 (blocks 0-33). Phase-1 value
|
# 0.68,0.32 = 45 of 65 blocks on the 3090 (blocks 0-44) - the maximum
|
||||||
# for the Q6_K swap (2026-08-17); the split will be re-maximized after
|
# for the Q6_K + q8_0/f16 KV config, measured 2026-08-17 phase 2:
|
||||||
# measuring the draft-model placement on the worker. With the Q6_K model
|
# - KV per FA layer = 784 MiB (K 1088 B/token + V 2048 B/token @262144)
|
||||||
# (layer pool 20334 MB) + q8_0/f16 KV (784 MiB per FA layer) the worker
|
# - weights(f) = 20334 MB layer pool x f (GGUF scan, exact); embd on
|
||||||
# frees ~4.7 GiB vs the old model at this split; the 0.53 boundary was
|
# the head side (RPC0 buffer observed 10429.88 MiB at f=0.53)
|
||||||
# provably safe under the old model and is comfortably safe here.
|
# - KV-per-device count = round(f x 66) layers -> 45 layers = 11 FA
|
||||||
TENSOR_SPLIT=0.53,0.47
|
# (3..43) -> 8624 MiB. 46 layers (f>=0.7045) still 11 FA but margin
|
||||||
|
# drops to ~0.4 GiB; 48 layers (f>=0.7197) adds FA block 47 -> KV
|
||||||
|
# 9408 -> OOM by ~0.85 GiB.
|
||||||
|
# Predicted: 13430 (weights) + 8624 (KV) + ~530 (RS) + 189 (compute) =
|
||||||
|
# ~22773 MiB -> ~0.7 GiB free. Draft model runs on ROCm0 only (measured
|
||||||
|
# phase 1: 1718 MiB, no worker impact).
|
||||||
|
TENSOR_SPLIT=0.68,0.32
|
||||||
|
|
||||||
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
|
# Context pool. 262144 total = 1 parallel slot x 262144 per slot
|
||||||
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).
|
# (n_ctx_seq = n_ctx / n_seq_max, llama-context.cpp:293).
|
||||||
|
|||||||
Reference in new issue
Block a user