frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep
Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B Q6_K_P, pp16384/tg512, fit-target 0). Key findings: - Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30); ubatch/batch do not affect decode at all. - Prefill + graph VRAM scale with ubatch; logical batch is irrelevant. - VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24. Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024. gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom. Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy; ncmoe32-ub768 (1.3GiB) if headroom matters most. Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
This commit is contained in:
@@ -52,6 +52,7 @@ DFlash draft, port 7770]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark that drove the Dankclaw llama.cpp deployment
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
|
||||
|
||||
## Speculative Decoding & Quantization
|
||||
|
||||
Reference in New Issue
Block a user