Frank
017d3704d2
The LLM fleet documentation (deployment runbook, model runbooks, benchmarks, research archive, retired-topology history) joins the project knowledge base as docs/LLM/. Wikilinks resolve by filename, so no link updates are needed for the move.
3.0 KiB
3.0 KiB
vLLM Index
Documentation Overview
This index contains all documentation for vLLM deployment on Dankserver and Dankclaw.
Hardware Guides
- Dankserver System Specs — NVIDIA GPU server specifications
- Dankclaw System Specs — AMD GPU server specifications
vLLM Architecture
- VLLM Overview — Complete vLLM documentation including installation, CLI, API, and key arguments
- RAM CPU Offloading in VLLM — CPU RAM offloading mechanisms, swap space, and memory configuration
- Hybrid CPU GPU Processing Dankserver — NVIDIA GPU with system RAM hybrid processing
- Shared Memory Architecture Dankclaw — AMD ROCm unified memory and APU configuration
Model Deployment Guides
- Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B FP8 deployment on NVIDIA GPU
- Running Qwen3.6 27B FP8 on Dankclaw — Qwen3.6-27B FP8 deployment on AMD GPU
Quick Reference
Installation Commands
# NVIDIA GPU
uv pip install vllm --torch-backend=auto
# AMD ROCm
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
# CPU
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
Server Start Commands
# Basic server
vllm serve Qwen/Qwen2.5-1.5B-Instruct
# With tensor parallelism
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
# With quantization
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8
# With GPU memory control
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space 8
Common API Endpoints
| Endpoint | Method | Description |
|---|---|---|
| /v1/models | GET | List available models |
| /v1/completions | POST | Text completion |
| /v1/chat/completions | POST | Chat completion |
| /health | GET | Health check |
| /metrics | GET | Prometheus metrics |
Key Parameters
| Parameter | Description | Typical Value |
|---|---|---|
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
| gpu_memory_utilization | Fraction of GPU memory for KV cache (default 0.92) | 0.7-0.95 |
| swap_space | Removed in current vLLM; use --kv-offloading-size |
n/a |
| max_model_len | Maximum context length | auto |
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
| max_num_seqs | Max sequences per batch | 128-512 |
Quantization Methods
| Method | GPU Support | Description |
|---|---|---|
| FP8 | Hopper, Ada (native); Ampere W8A16 via Marlin; RDNA via emulation | FP8 E4M3 inference |
| INT8 | Most GPUs | Weight-only INT8 |
| GPTQ | Most GPUs (NVIDIA; limited on ROCm) | Post-training GPTQ |
| AWQ | Turing+ (NVIDIA) | Activation-aware weight quantization |
| GGUF | CPU, GPU | General GPU unification format |