Frank
35c44c4123
Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index, Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0 draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16). deploy.sh now maps dankserver to frank-default-small and symlinks its unit. models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
69 lines
2.2 KiB
Markdown
69 lines
2.2 KiB
Markdown
# Dankserver System Specs
|
|
|
|
## Server Overview
|
|
|
|
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.
|
|
|
|
## Hardware Specifications
|
|
|
|
### CPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Architecture | x86_64 |
|
|
| Vendor | AuthenticAMD |
|
|
| Model | AMD Ryzen 7 7700X |
|
|
| Cores | 8 (1 socket) |
|
|
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
|
|
|
### GPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Model | NVIDIA GeForce RTX 3090 |
|
|
| VRAM | 24576 MiB (24 GB) |
|
|
| Driver | 595.84 |
|
|
| CUDA Version | 13.2 |
|
|
| CUDA Toolkit | 12.0 (nvcc) |
|
|
|
|
### Memory
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| System RAM | 62 GiB |
|
|
| Swap | 127 GiB |
|
|
| /dev/shm | 32 GiB |
|
|
|
|
### Storage
|
|
|
|
| Filesystem | Size | Used | Avail | Use% |
|
|
|------------|------|------|-------|------|
|
|
| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
|
|
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
|
|
|
|
### OS and Software
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
|
|
| Kernel | 7.0.0-28-generic |
|
|
| Python | 3.12.3 |
|
|
|
|
### Active Services
|
|
|
|
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped).
|
|
|
|
## Hardware Relevance
|
|
|
|
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU.
|
|
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
|
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
|
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
|
|
|
## See Also
|
|
|
|
- [[VLLM Overview]]
|
|
- [[Hybrid CPU GPU Processing Dankserver]]
|
|
- [[RAM CPU Offloading in VLLM]]
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|