docs+deploy: Dankserver now llama.cpp frank-default-small (HauhauCS Q6_K_P + DFlash)

Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index,
Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream
llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0
draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16).
deploy.sh now maps dankserver to frank-default-small and symlinks its unit.
models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
This commit is contained in:
Frank committed 2026-08-11 01:18:29 -07:00
1 parent 6a86a9df93
commit 35c44c4123
8 files changed
+34 -32

No files matched your search

+6 -7
View File
@@ -51,15 +51,14 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services
A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies approximately 22 GB of the RTX 3090 VRAM (`nvidia-smi` reports `VLLM::EngineCore` at 22164 MiB).
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
## vLLM Relevance
## Hardware Relevance
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
- 62 GB system RAM provides ample swap space for KV cache offloading.
- 127 GB swap supports large `--swap-space` values for CPU KV cache paging.
- Root disk has approximately 140 GB free. Store models on /mnt/entropy.
- CUDA 12.0 and Python 3.12.3 are compatible with vLLM.
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
## See Also