dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: polish pass — fix stale names, duplicates, branding, and README

Browse Source 
- Normalize 'VLLM' to 'vLLM' in prose across 6 docs (32 occurrences; wikilink
  targets untouched)
- Remove duplicated Environment Variables block in 27B Dankclaw doc
- Update stale model refs Qwen3-30B-A3B -> Qwen3.6-35B-A3B in example commands
- Fix VLLM Index: -- separators to em-dashes; correct model names on links
- README: document active services (ROCmFPX llama-server on Dankclaw, vLLM
  AWQ on Dankserver), replace broken [[docs/README.md]] link with docs/index.md
This commit is contained in:
Frank
2026-08-10 21:12:23 -07:00
parent 993b6af576
commit f650fff0aa
8 changed files with 62 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+21 -8
View File
@@ -1,26 +1,39 @@
# vLLM Scripts
Scripts for running large language models via vLLM on two local machines.
Scripts for running LLM inference on two local machines.
## Targets
| Host | User | Codename | Architecture |
|------|------|----------|-------------|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
## Active Services
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| llama-server | dankclaw | ROCmFPX llama.cpp | Ling-3.0-flash ROCmFP4 MTP | 8080 |
| vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 |
## Models
| Model | Target | Format |
|-------|--------|--------|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
| Ling-3.0-flash ROCmFP4-MTP | dankclaw | ROCmFP4 GGUF |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
## Structure
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
- `scripts/` — Executable vLLM launch scripts (coming soon)
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-vllm-qwen36-dflash.sh, validate-vllm.sh, systemd unit files)
## Deployment
See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.
## Knowledge Base
See [[docs/README.md]] for the project knowledge base index.
See [docs/index.md](docs/index.md) for the project knowledge base index.
docs/Dankserver System Specs.md
+1 -1
View File
@@ -53,7 +53,7 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies approximately 22 GB of the RTX 3090 VRAM (`nvidia-smi` reports `VLLM::EngineCore` at 22164 MiB).
## VLLM Relevance
## vLLM Relevance
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
- 62 GB system RAM provides ample swap space for KV cache offloading.
docs/Hybrid CPU GPU Processing Dankserver.md
+5 -5
View File
@@ -2,13 +2,13 @@
## Overview
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. VLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. vLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
## Architecture
### Model Weights
Model weights stay on GPU. VLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
Model weights stay on GPU. vLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
### KV Cache
@@ -70,7 +70,7 @@ vllm serve meta-llama/Llama-3.1-70B-Instruct \
Use FP8 KV cache for Ampere and newer GPUs:
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16
@@ -177,7 +177,7 @@ htop
### Single GPU, Medium Model
```
vllm serve Qwen/Qwen3-30B-A3B \
vllm serve Qwen/Qwen3.6-35B-A3B \
--gpu-memory-utilization 0.85 \
--swap-space 16 \
--max-num-batched-tokens 4096 \
@@ -187,7 +187,7 @@ vllm serve Qwen/Qwen3-30B-A3B \
### Single GPU, FP8 Model
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16 \
docs/RAM CPU Offloading in VLLM.md
+6 -6
View File
@@ -1,8 +1,8 @@
# RAM CPU Offloading in VLLM
# RAM CPU Offloading in vLLM
## Overview
VLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. VLLM does not offload model weights to CPU. Instead, VLLM moves KV cache data between GPU and CPU.
vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU.
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
@@ -167,10 +167,10 @@ If you see frequent preemptions:
| Variable | Description | Default |
|----------|-------------|---------|
| VLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| VLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| VLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| VLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
## See Also
docs/Running Qwen3.6 27B FP8 on Dankclaw.md
-9
View File
@@ -76,15 +76,6 @@ export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface
```
## Environment Variables
Set these for ROCm on Strix Halo:
```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface
```
## vLLM Serve Commands
### Text-Only (Recommended for Memory Savings)
docs/Shared Memory Architecture Dankclaw.md
+6 -6
View File
@@ -13,12 +13,12 @@ In unified memory:
1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to VLLM
4. The total available memory appears as GPU memory to vLLM
5. Memory pages move between CPU and GPU transparently
### Implications for VLLM
### Implications for vLLM
- VLLM sees the full memory as GPU memory
- vLLM sees the full memory as GPU memory
- No explicit swap_space parameter needed for memory extension
- The GPU driver handles memory paging automatically
- Performance depends on memory bandwidth and latency
@@ -206,7 +206,7 @@ For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native
### Basic APU Deployment
```
vllm serve Qwen/Qwen3-30B-A3B \
vllm serve Qwen/Qwen3.6-35B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
@@ -217,7 +217,7 @@ vllm serve Qwen/Qwen3-30B-A3B \
### With FP8 Model
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--kv-cache-dtype fp8 \
@@ -227,7 +227,7 @@ vllm serve Qwen/Qwen3-30B-A3B-FP8 \
### With Long Context
```
vllm serve Qwen/Qwen3-30B-A3B \
vllm serve Qwen/Qwen3.6-35B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
docs/VLLM Index.md
+11 -11
View File
@@ -1,25 +1,25 @@
# VLLM Index
# vLLM Index
## Documentation Overview
This index contains all documentation for VLLM deployment on Dankserver and Dankclaw.
This index contains all documentation for vLLM deployment on Dankserver and Dankclaw.
## Hardware Guides
- [[Dankserver System Specs]] -- NVIDIA GPU server specifications
- [[Dankclaw System Specs]] -- AMD GPU server specifications
- [[Dankserver System Specs]] — NVIDIA GPU server specifications
- [[Dankclaw System Specs]] — AMD GPU server specifications
## VLLM Architecture
## vLLM Architecture
- [[VLLM Overview]] -- Complete VLLM documentation including installation, CLI, API, and key arguments
- [[RAM CPU Offloading in VLLM]] -- CPU RAM offloading mechanisms, swap space, and memory configuration
- [[Hybrid CPU GPU Processing Dankserver]] -- NVIDIA GPU with system RAM hybrid processing
- [[Shared Memory Architecture Dankclaw]] -- AMD ROCm unified memory and APU configuration
- [[VLLM Overview]] — Complete vLLM documentation including installation, CLI, API, and key arguments
- [[RAM CPU Offloading in VLLM]] — CPU RAM offloading mechanisms, swap space, and memory configuration
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU with system RAM hybrid processing
- [[Shared Memory Architecture Dankclaw]] — AMD ROCm unified memory and APU configuration
## Model Deployment Guides
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] -- Qwen3-30B-A3B FP8 deployment on NVIDIA GPU
- [[Running Qwen3.6 27B FP8 on Dankclaw]] -- Qwen3-32B FP8 deployment on AMD GPU
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B FP8 deployment on NVIDIA GPU
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B FP8 deployment on AMD GPU
## Quick Reference
docs/VLLM Overview.md
+12 -12
View File
@@ -1,10 +1,10 @@
# VLLM Overview
# vLLM Overview
## What Is VLLM
## What Is vLLM
VLLM is a fast and easy library for large language model inference and serving. You can use VLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. VLLM supports over 200 model architectures from Hugging Face.
vLLM is a fast and easy library for large language model inference and serving. You can use vLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. vLLM supports over 200 model architectures from Hugging Face.
VLLM is fast with these features:
vLLM is fast with these features:
- PagedAttention for efficient key and value memory management
- Continuous batching of incoming requests
@@ -15,7 +15,7 @@ VLLM is fast with these features:
- Automatic kernel generation with torch.compile
- Disaggregated prefill, decode, and encode
VLLM is flexible with these features:
vLLM is flexible with these features:
- Seamless integration with Hugging Face models
- High-throughput serving with parallel sampling and beam search
@@ -28,7 +28,7 @@ VLLM is flexible with these features:
## Installation
You can install VLLM with uv (recommended) or pip:
You can install vLLM with uv (recommended) or pip:
```
uv pip install vllm
@@ -174,15 +174,15 @@ curl http://localhost:8000/v1/chat/completions \
## Architecture
VLLM uses a multi-process architecture:
vLLM uses a multi-process architecture:
- API server process handles HTTP requests, tokenization, and input processing
- Engine core process runs the scheduler and coordinates GPU workers
- One GPU worker process per GPU executes model forward passes
VLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.
vLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.
VLLM supports multiple attention backends:
vLLM supports multiple attention backends:
- FlashAttention for NVIDIA GPUs
- FlashInfer for NVIDIA GPUs
@@ -191,7 +191,7 @@ VLLM supports multiple attention backends:
## Supported Models
VLLM supports over 200 model architectures including:
vLLM supports over 200 model architectures including:
- Decoder-only LLMs (Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
@@ -202,7 +202,7 @@ VLLM supports over 200 model architectures including:
## Quantization
VLLM supports these quantization methods:
vLLM supports these quantization methods:
- FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
- MXFP8/MXFP4 for Hopper GPUs
@@ -220,7 +220,7 @@ AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not na
## Parallelism
VLLM supports multiple parallelism strategies:
vLLM supports multiple parallelism strategies:
- Tensor Parallelism (TP) shards model parameters across GPUs
- Pipeline Parallelism (PP) distributes model layers across GPUs