feat: initialize vLLM scripts knowledge base
- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism - VLLM Index: quick reference with commands, endpoints, parameters - Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04 - Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1 - RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption - Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration - Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup - Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide - Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
This commit is contained in:
commit
39db415348
11 files changed
+1577
No files matched your search
@@ -0,0 +1,68 @@
|
||||
# Dankserver System Specs
|
||||
|
||||
## Server Overview
|
||||
|
||||
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.
|
||||
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Architecture | x86_64 |
|
||||
| Vendor | AuthenticAMD |
|
||||
| Cores | 8 (1 socket) |
|
||||
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
||||
|
||||
### GPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | NVIDIA GeForce RTX 3090 |
|
||||
| VRAM | 24576 MiB (24 GB) |
|
||||
| Driver | 595.84 |
|
||||
| CUDA Version | 13.2 |
|
||||
| CUDA Toolkit | 12.0 (nvcc) |
|
||||
|
||||
### Memory
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| System RAM | 62 GiB |
|
||||
| Swap | 127 GiB |
|
||||
| /dev/shm | 32 GiB |
|
||||
|
||||
### Storage
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 915G | 849G | 20G | 98% |
|
||||
| /mnt/entropy (dm-0) | 91T | 86T | 977G | 99% |
|
||||
|
||||
### OS and Software
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
|
||||
| Kernel | 7.0.0-28-generic |
|
||||
| Python | 3.12.3 |
|
||||
|
||||
### Active Services
|
||||
|
||||
A llama-server process occupies approximately 20 GB of the RTX 3090 VRAM.
|
||||
|
||||
## VLLM Relevance
|
||||
|
||||
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
|
||||
- 62 GB system RAM provides ample swap space for KV cache offloading.
|
||||
- 127 GB swap supports large `--swap-space` values for CPU KV cache paging.
|
||||
- Root disk has only 20 GB free. Store models on /mnt/entropy.
|
||||
- CUDA 12.0 and Python 3.12.3 are compatible with vLLM.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
Reference in new issue
Block a user