frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep

Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B
Q6_K_P, pp16384/tg512, fit-target 0). Key findings:
- Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30);
  ubatch/batch do not affect decode at all.
- Prefill + graph VRAM scale with ubatch; logical batch is irrelevant.
- VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the
  historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24.

Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024.
  gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom.
Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy;
  ncmoe32-ub768 (1.3GiB) if headroom matters most.
Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
This commit is contained in:
Frank committed 2026-08-13 22:56:24 -07:00
1 parent 0d09756430
commit 3c2db64a78
6 files changed
+145 -11

No files matched your search

+2 -2
View File
@@ -32,10 +32,10 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
: "${N_CPU_MOE:=32}" # 32 of 40 layers: MoE expert weights stay on CPU; 8 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${UBATCH_SIZE:=1024}"
: "${CTX_CHECKPOINTS:=0}"
: "${NO_MMAP:=0}"
: "${TEMPERATURE:=1.0}"