Files

T

Frank 39db415348 feat: initialize vLLM scripts knowledge base

- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism
- VLLM Index: quick reference with commands, endpoints, parameters
- Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04
- Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1
- RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption
- Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration
- Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup
- Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide
- Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide

2026-08-05 23:49:02 -07:00

1.6 KiB

Raw Blame History

Dankserver System Specs

Server Overview

Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.

Hardware Specifications

CPU

Property	Value
Architecture	x86_64
Vendor	AuthenticAMD
Cores	8 (1 socket)
NUMA Nodes	1 (node 0, 63503 MB)

GPU

Property	Value
Model	NVIDIA GeForce RTX 3090
VRAM	24576 MiB (24 GB)
Driver	595.84
CUDA Version	13.2
CUDA Toolkit	12.0 (nvcc)

Memory

Property	Value
System RAM	62 GiB
Swap	127 GiB
/dev/shm	32 GiB

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	915G	849G	20G	98%
/mnt/entropy (dm-0)	91T	86T	977G	99%

OS and Software

Property	Value
OS	Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel	7.0.0-28-generic
Python	3.12.3

Active Services

A llama-server process occupies approximately 20 GB of the RTX 3090 VRAM.

VLLM Relevance

24 GB VRAM limits model size. Use FP8 quantization for large models.
62 GB system RAM provides ample swap space for KV cache offloading.
127 GB swap supports large --swap-space values for CPU KV cache paging.
Root disk has only 20 GB free. Store models on /mnt/entropy.
CUDA 12.0 and Python 3.12.3 are compatible with vLLM.

1.6 KiB Raw Blame History