dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: prefill-max ncmoe 35 / ubatch 3072

Browse Source 
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
  ~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
  decode cost. Docs synced (benchmark Chosen State, deployment, specs).
This commit is contained in:
Frank committed 2026-08-13 23:07:30 -07:00
1 parent 2b4e58a926
commit 01eb957f90
5 files changed
+28 -23
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/deployment.md
+1 -1
View File
@@ -143,7 +143,7 @@ The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B u
### Dankserver — frank-default-small
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **32 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 32`); 8 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 1024, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **35 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 35`); 5 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.