Frank
4462aa992c
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU (replaces --n-cpu-moe), Qwen-Sharp chat template. - frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0 ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head (--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the retired PromptForge .pfs sidecar lanes. - Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via --chat-template-file on both services. - Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages; update index, deployment, and both system specs; models.json names. Smoke-tested both models on the real hardware on scratch ports before commit: model loads, MTP draft attaches, Sharp template applies, chat completion returns.
77 lines
3.1 KiB
Markdown
77 lines
3.1 KiB
Markdown
# Dankserver System Specs
|
|
|
|
## Server Overview
|
|
|
|
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM.
|
|
|
|
## Hardware Specifications
|
|
|
|
### CPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Architecture | x86_64 |
|
|
| Vendor | AuthenticAMD |
|
|
| Model | AMD Ryzen 7 7700X |
|
|
| Cores | 8 (1 socket) |
|
|
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
|
|
|
### GPU
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Model | NVIDIA GeForce RTX 3090 |
|
|
| VRAM | 24576 MiB (24 GB) |
|
|
| Driver | 595.84 |
|
|
| CUDA Version | 13.2 |
|
|
| CUDA Toolkit | 12.0 (nvcc) |
|
|
|
|
### Memory
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| System RAM | 62 GiB |
|
|
| Swap | 127 GiB |
|
|
| /dev/shm | 32 GiB |
|
|
|
|
### Storage
|
|
|
|
| Filesystem | Size | Used | Avail | Use% |
|
|
|------------|------|------|-------|------|
|
|
| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
|
|
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
|
|
|
|
### Network
|
|
|
|
| Interface | Type | Speed | IP | Notes |
|
|
|-----------|------|-------|----|-------|
|
|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. |
|
|
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). |
|
|
| wlp8s0 | WiFi | - | - | Unused (carrier down). |
|
|
| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. |
|
|
|
|
### OS and Software
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
|
| Kernel | 7.0.0-30-generic |
|
|
| Python | 3.14.4 |
|
|
|
|
### Active Services
|
|
|
|
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q8_0) on port 7770, with MoE expert weights held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
|
|
|
|
## Hardware Relevance
|
|
|
|
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load.
|
|
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
|
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
|
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
|
|
|
## See Also
|
|
|
|
- [[VLLM Overview]]
|
|
- [[Hybrid CPU GPU Processing Dankserver]]
|
|
- [[RAM CPU Offloading in VLLM]]
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] |