dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record dual-f16 KV refit on frank-default-small (2026-08-14)

Browse Source
This commit is contained in:
Frank committed 2026-08-14 12:59:10 -07:00
1 parent f96aa2b692
commit d72092b2c7
4 files changed
+26 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/deployment.md
+1 -1
View File
@@ -143,7 +143,7 @@ The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B u
### Dankserver — frank-default-small
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).