Tune small model expert placement

This commit is contained in:
Frank committed 2026-08-25 00:31:05 -07:00
1 parent af6ffeeb64
commit 7a17984dbc
5 files changed
+16 -16

No files matched your search

+3 -3
View File
@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 36 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 37 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.76 GiB of CPU expert tensors with room to spare.
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).