Frank
136ee080c9
- Ethernet Bandwidth Math: new section 0 with live measurements (iperf 281-294 MB/s, ping RTT both paths, prefill 457.9 t/s at 2.5GbE, per-ubatch wire 20.5 MB, 98k KV snapshot 3.45 GB blocking the slot 13.6 s, validated link-speed model: 386/458/505 t/s) - System specs: NIC tables for both boxes; the 1 GbE USB-C adapter on the claw was silently routing the RPC path (unplugged 2026-08-17) - PP Deployment: fix hidden-state size (10 KiB/direction, H=5120) and RTT-vs-bandwidth claim (decode RTT-bound; prefill+snapshot BW-bound)
77 lines
3.1 KiB
Markdown
77 lines
3.1 KiB
Markdown
# Dankserver System Specs
|
|
|
|
## Server Overview
|
|
|
|
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.
|
|
|
|
## Hardware Specifications
|
|
|
|
### CPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Architecture | x86_64 |
|
|
| Vendor | AuthenticAMD |
|
|
| Model | AMD Ryzen 7 7700X |
|
|
| Cores | 8 (1 socket) |
|
|
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
|
|
|
### GPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Model | NVIDIA GeForce RTX 3090 |
|
|
| VRAM | 24576 MiB (24 GB) |
|
|
| Driver | 595.84 |
|
|
| CUDA Version | 13.2 |
|
|
| CUDA Toolkit | 12.0 (nvcc) |
|
|
|
|
### Memory
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| System RAM | 62 GiB |
|
|
| Swap | 127 GiB |
|
|
| /dev/shm | 32 GiB |
|
|
|
|
### Storage
|
|
|
|
| Filesystem | Size | Used | Avail | Use% |
|
|
|------------|------|------|-------|------|
|
|
| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
|
|
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
|
|
|
|
### Network
|
|
|
|
| Interface | Type | Speed | IP | Notes |
|
|
|-----------|------|-------|----|-------|
|
|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. |
|
|
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). |
|
|
| wlp8s0 | WiFi | - | - | Unused (carrier down). |
|
|
| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. |
|
|
|
|
### OS and Software
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
|
|
| Kernel | 7.0.0-28-generic |
|
|
| Python | 3.12.3 |
|
|
|
|
### Active Services
|
|
|
|
The active service on Dankserver is `frank-pp-worker.service`: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (`frank-default-small`) is retired (2026-08-16) and disabled (2026-08-17). See [[Qwen3.8 PP Deployment]].
|
|
|
|
## Hardware Relevance
|
|
|
|
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load.
|
|
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
|
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
|
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
|
|
|
## See Also
|
|
|
|
- [[VLLM Overview]]
|
|
- [[Hybrid CPU GPU Processing Dankserver]]
|
|
- [[RAM CPU Offloading in VLLM]]
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] |