diff --git a/docs/Dankserver System Specs.md b/docs/Dankserver System Specs.md index 56065d8..c4a50c2 100644 --- a/docs/Dankserver System Specs.md +++ b/docs/Dankserver System Specs.md @@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0 ### Active Services -The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 14 GB of the 24 GB VRAM and about 32 GB of RAM. +The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 200000 context each (pool 800000). Of the 40 layers' MoE expert weights, **32 sit on CPU** (`--n-cpu-moe 32`) while the remaining **8 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`q8_0`, V=`f16`. Batch 8192 / ubatch 2048, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~23.5 GB VRAM used; under a 4-stream 300-token burst peaks ~23.7 GB, leaving **~870 MiB** headroom. Idle RAM ~32 GB of 62 GB. ## Hardware Relevance -- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context plus batch 3072/1536 activation buffers, the non-expert weights and KV cache consume about 14 GB, leaving about 10 GB of headroom. MoE experts stay on CPU; `--no-mmap` preloads expert tensors into RAM, eliminating decode page faults (uses ~32 GB of the 62 GB available). +- 24 GB VRAM is almost fully saturated with this config. At idle ~23.5 GB used; under a 4-stream 300-token burst peaks at ~23.7 GB, leaving **~870 MiB** headroom. The 8 layers' expert weights back on GPU account for the tight margin. If headroom proves too thin for production bursts, dial back `--n-cpu-moe` to 36 (only 4 layers on GPU, ~1.7 GiB more headroom) by editing `scripts/frank-default-small.env`. - 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load). - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). @@ -65,4 +65,4 @@ The active `frank-default-small` service (llama.cpp, `b10108+`) serves the Hauha - [[VLLM Overview]] - [[Hybrid CPU GPU Processing Dankserver]] - [[RAM CPU Offloading in VLLM]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] +- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] \ No newline at end of file diff --git a/docs/deployment.md b/docs/deployment.md index e55b917..68c6f00 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -104,7 +104,7 @@ The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash C ### Dankserver — frank-default-small -The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE expert weights stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. +The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 200000 context each (pool 800000), and mmproj vision. **32 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 32`); 8 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`q8_0`, V=`f16`). Batch 8192 / ubatch 2048, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. > Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.