docs+deploy: Dankserver now llama.cpp frank-default-small (HauhauCS Q6_K_P + DFlash)
Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index, Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0 draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16). deploy.sh now maps dankserver to frank-default-small and symlinks its unit. models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
This commit is contained in:
1 parent
6a86a9df93
commit
35c44c4123
8 files changed
+34
-32
No files matched your search
@@ -51,15 +51,14 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies approximately 22 GB of the RTX 3090 VRAM (`nvidia-smi` reports `VLLM::EngineCore` at 22164 MiB).
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
|
||||
|
||||
## vLLM Relevance
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
|
||||
- 62 GB system RAM provides ample swap space for KV cache offloading.
|
||||
- 127 GB swap supports large `--swap-space` values for CPU KV cache paging.
|
||||
- Root disk has approximately 140 GB free. Store models on /mnt/entropy.
|
||||
- CUDA 12.0 and Python 3.12.3 are compatible with vLLM.
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user