docs: frank-default-small now 200000 ctx/stream, batch 3072/1536, --no-mmap

Sync deployment.md, Dankserver specs, DFlash doc to the tuned config:
ctx reduced to 200000 per stream (pool 600000) to free KV VRAM for the larger
batch; --no-mmap eager RAM load; ctx-checkpoints 4; ~20GB VRAM / 32GB RAM.
models.json contextWindow -> 200000.
This commit is contained in:
Frank
2026-08-11 01:52:12 -07:00
parent 9971f83e8b
commit 5c9e812221
3 changed files with 4 additions and 4 deletions
+1 -1
View File
@@ -115,7 +115,7 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 262144 context, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 200000 context each, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
+2 -2
View File
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 20 GB of the 24 GB VRAM and about 32 GB of RAM.
## Hardware Relevance
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context, the full-attention KV cache plus non-expert weights and the batch 3072/1536 activation buffers consume about 20 GB, leaving about 4 GB of headroom. MoE experts must stay on CPU.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
+1 -1
View File
@@ -104,7 +104,7 @@ The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash C
### Dankserver — frank-default-small
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 262144 context each, and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.