serverAppConfigs/docs/LLM/VLLM Index.md

vLLM Index

RETIRED (2026-08-18). The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See ciru-activefpx-promptforge.

Documentation Overview

This index contains all documentation for vLLM deployment on Dankserver and Dankclaw.

Hardware Guides

• Dankserver System Specs — NVIDIA GPU server specifications
• Dankclaw System Specs — AMD GPU server specifications

vLLM Architecture

• VLLM Overview — Complete vLLM documentation including installation, CLI, API, and key arguments
• RAM CPU Offloading in VLLM — CPU RAM offloading mechanisms, swap space, and memory configuration
• Hybrid CPU GPU Processing Dankserver — NVIDIA GPU with system RAM hybrid processing
• Shared Memory Architecture Dankclaw — AMD ROCm unified memory and APU configuration

Model Deployment Guides

• Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B FP8 deployment on NVIDIA GPU
• Running Qwen3.6 27B FP8 on Dankclaw — Qwen3.6-27B FP8 deployment on AMD GPU

Quick Reference

Installation Commands

# NVIDIA GPU
uv pip install vllm --torch-backend=auto

# AMD ROCm
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

# CPU
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/

Server Start Commands

# Basic server
vllm serve Qwen/Qwen2.5-1.5B-Instruct

# With tensor parallelism
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

# With quantization
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8

# With GPU memory control
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space 8

Common API Endpoints

Endpoint	Method	Description
/v1/models	GET	List available models
/v1/completions	POST	Text completion
/v1/chat/completions	POST	Chat completion
/health	GET	Health check
/metrics	GET	Prometheus metrics

Key Parameters

Parameter	Description	Typical Value
tensor_parallel_size	Number of GPUs for model sharding	1-8
gpu_memory_utilization	Fraction of GPU memory for KV cache (default 0.92)	0.7-0.95
swap_space	Removed in current vLLM; use --kv-offloading-size	n/a
max_model_len	Maximum context length	auto
max_num_batched_tokens	Max tokens per batch	2048-8192
max_num_seqs	Max sequences per batch	128-512

Quantization Methods

Method	GPU Support	Description
FP8	Hopper, Ada (native); Ampere W8A16 via Marlin; RDNA via emulation	FP8 E4M3 inference
INT8	Most GPUs	Weight-only INT8
GPTQ	Most GPUs (NVIDIA; limited on ROCm)	Post-training GPTQ
AWQ	Turing+ (NVIDIA)	Activation-aware weight quantization
GGUF	CPU, GPU	General GPU unification format

See Also

• VLLM Overview
• RAM CPU Offloading in VLLM
• Hybrid CPU GPU Processing Dankserver
• Shared Memory Architecture Dankclaw
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw