frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep

Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B
Q6_K_P, pp16384/tg512, fit-target 0). Key findings:
- Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30);
  ubatch/batch do not affect decode at all.
- Prefill + graph VRAM scale with ubatch; logical batch is irrelevant.
- VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the
  historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24.

Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024.
  gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom.
Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy;
  ncmoe32-ub768 (1.3GiB) if headroom matters most.
Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.

scripts/frank-default-small.env

@@ -8,15 +8,17 @@ PORT=7770
 # (Increased from 3×200k to add a 4th parallel stream.)
 CTX_SIZE=800000
 PARALLEL=4
-# Spec: ngram-mod. n_min=48, n_max=64, n_match=24 (defaults). 5 expert
-# layers back on GPU; 35 on CPU. Batch 8192 / ubatch 2048.
+# Spec: ngram-mod. n_min=48, n_max=64, n_match=24 (defaults). 8 expert
+# layers back on GPU; 32 on CPU. Batch 8192 / ubatch 1024.
 SPEC_TYPE=ngram-mod
 NGRAM_MOD_N_MIN=48
 NGRAM_MOD_N_MAX=64
 NGRAM_MOD_N_MATCH=24
-# Expert offload: 35 of 40 layers keep MoE weights on CPU; remaining
-# 5 layers have their expert weights resident on GPU for faster routing.
-N_CPU_MOE=35
+# Expert offload: 32 of 40 layers keep MoE weights on CPU; remaining
+# 8 layers have their expert weights resident on GPU for faster routing.
+# Benchmark-chosen (see frank-default-small-benchmark.md): max decode at
+# the headroom that stays above the historical OOM boundary.
+N_CPU_MOE=32
 N_GPU_LAYERS=all
 # KV cache on GPU, K quantized to q8_0, V stays f16 (user-locked decision).
 CACHE_TYPE_K=q8_0
@@ -25,8 +27,10 @@ CACHE_TYPE_V=f16
 # larger batch/ubatch (leaves headroom for ctx-checkpoints operation).
 CTX_CHECKPOINTS=4
 # Larger batches / ubatch for big prefill workloads.
+# ubatch 1024 (benchmark-chosen): decode is flat vs ubatch; 1024 frees
+# compute-graph VRAM to afford ncmoe 32 while keeping prefill ~766 t/s.
 BATCH_SIZE=8192
-UBATCH_SIZE=2048
+UBATCH_SIZE=1024
 # Load the model into RAM rather than lazy-mapping it. The MoE experts live on
 # CPU, so eager RAM load avoids page faults during decode (uses ~31GB RAM).
 NO_MMAP=1

scripts/serve-llama-frank-small.sh

@@ -32,10 +32,10 @@ fi
 : "${NGRAM_MOD_N_MATCH:=24}"
 : "${CACHE_TYPE_K:=q8_0}"
 : "${CACHE_TYPE_V:=f16}"
-: "${N_CPU_MOE:=35}"     # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
+: "${N_CPU_MOE:=32}"     # 32 of 40 layers: MoE expert weights stay on CPU; 8 on GPU
 : "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
 : "${BATCH_SIZE:=2048}"
-: "${UBATCH_SIZE:=512}"
+: "${UBATCH_SIZE:=1024}"
 : "${CTX_CHECKPOINTS:=0}"
 : "${NO_MMAP:=0}"
 : "${TEMPERATURE:=1.0}"