frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots triple the KV, so expert blocks 27-34 are pinned back to CPU (freed ~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model, split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md: drop the scratch-port rule (test/bench processes cannot share the single GPU with the running service anyway). Env, serve script, bats, docs updated.
This commit is contained in:
Frank committed
1 parent
8c41c821c4
commit
04a8fb7806
5 files changed
+84
-68
No files matched your search
@@ -5,8 +5,8 @@
|
||||
# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
|
||||
# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
|
||||
# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
|
||||
# KV cache quantized to iq5_k with Hadamard transforms on both K and V.
|
||||
# Split mode none (single GPU; graph would require 2+ devices).
|
||||
# KV cache K q8_0 / V q5_0 with Hadamard transforms on both (see the KV
|
||||
# section below). Split mode none (single GPU; graph would require 2+).
|
||||
#
|
||||
# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
|
||||
# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
|
||||
@@ -43,40 +43,41 @@ ALIAS=frank-default-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative
|
||||
# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE
|
||||
# =none) unless a user asks to re-enable ngram-mod/MTP.
|
||||
# Three parallel slots, each with 160000 ctx. With PARALLEL=3 ik_llama
|
||||
# limits speculative decoding to a single slot, so SPEC stays none (MTP is
|
||||
# also disabled). The 3 slots triple the KV cache, so enough expert weights
|
||||
# are CPU-pinned for the cache to fit on the 24 GB card.
|
||||
CTX_SIZE=160000
|
||||
PARALLEL=1
|
||||
PARALLEL=3
|
||||
|
||||
# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays
|
||||
# off for the low-parameter MoE (3B active of 35B). MTP is also disabled.
|
||||
# Speculative decoding: DISABLED. ik_llama limits spec to a single slot; at
|
||||
# PARALLEL=3 a configured spec stage hard-exits at load. The low-parameter
|
||||
# MoE (3B active of 35B) does not need it anyway. MTP is also disabled.
|
||||
SPEC_TYPE=none
|
||||
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
|
||||
# layer audit found 630 MiB of expert tensors per block. Keep every
|
||||
# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
|
||||
# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
|
||||
# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
|
||||
# CPU buft name on this host.
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q8_0
|
||||
# audit found ~1088 MiB of expert tensors per block (4 tensor × 272 MiB). Keep
|
||||
# every final/output-adjacent block 35-40 on CUDA; full-attention blocks 3,7,
|
||||
# 11,15,19,23 have no exps and are untouched by this regex. All other expert
|
||||
# tensors are pinned to CPU. Buffer type "CPU" is the CPU buft on this host.
|
||||
#
|
||||
# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
|
||||
# use the ~5.4 GiB GPU headroom for lower latency; then 28,29 (~1.26 GiB)
|
||||
# for the fused-MoE run. Left ~1.6 GiB headroom. The remaining larger-CPU
|
||||
# range below those stays on CPU.
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
# 2026-08-26: PARALLEL=3 (CTX 160000) triples the KV cache, so the blocks
|
||||
# that were moved back to CUDA for fused-MoE (27-34) are pinned to CPU again
|
||||
# to free GPU. Blocks 35-40 stay on CUDA. If it boot-loops, move the next
|
||||
# lower block into the CPU regex one at a time.
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
|
||||
# mean "load as many layers as possible". Expert exps are pinned to CPU by
|
||||
# OVERRIDE_EXPERT below.
|
||||
N_GPU_LAYERS=999
|
||||
|
||||
# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both
|
||||
# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build
|
||||
# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we
|
||||
# verified works. Flash-attention is on by default.
|
||||
CACHE_TYPE_K=q6_0
|
||||
CACHE_TYPE_V=q6_0
|
||||
# KV cache on GPU: K at q8_0 (8-bit) and V at q5_0 (5-bit), each with
|
||||
# Hadamard transforms. The V cache at lower precision than K is a common
|
||||
# asymmetric choice; q8_0 worked (plain q8_KV still crashes here, ggml.c
|
||||
# :5427). Flash-attention is on by default.
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q5_0
|
||||
K_CACHE_HADAMARD=1
|
||||
V_CACHE_HADAMARD=1
|
||||
|
||||
|
||||
Reference in new issue
Block a user