feat: initialize vLLM scripts knowledge base

- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism - VLLM Index: quick reference with commands, endpoints, parameters - Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04 - Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1 - RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption - Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration - Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup - Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide - Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
2026-08-05 23:49:02 -07:00
commit 39db415348
@@ -0,0 +1,265 @@
+# Shared Memory Architecture Dankclaw
+
+## Overview
+
+Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.
+
+## Unified Memory Architecture
+
+### How Unified Memory Works
+
+In unified memory:
+
+1. CPU and GPU share one physical memory pool
+2. The OS and GPU driver manage memory allocation
+3. There is no separate VRAM or system RAM
+4. The total available memory appears as GPU memory to VLLM
+5. Memory pages move between CPU and GPU transparently
+
+### Implications for VLLM
+
+- VLLM sees the full memory as GPU memory
+- No explicit swap_space parameter needed for memory extension
+- The GPU driver handles memory paging automatically
+- Performance depends on memory bandwidth and latency
+- Shared memory means CPU and GPU compete for the same pool
+
+## ROCm Configuration
+
+### ROCm Version
+
+vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw should have ROCm 7.0 or later.
+
+Check ROCm version:
+
+```
+rocminfo
+```
+
+### GPU Architecture
+
+Strix Halo uses the gfx115x architecture. Check with:
+
+```
+rocminfo | grep gfx
+```
+
+### Installation
+
+Install VLLM for ROCm with Python 3.12:
+
+```
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
+```
+
+### Attention Backend
+
+Choose the best attention backend for Strix Halo:
+
+```
+vllm serve model \
+    --attention-backend ROCM_AITER_UNIFIED_ATTN
+```
+
+Available backends for ROCm:
+
+| Backend | Description | Best For |
+|---------|-------------|----------|
+| TRITON_ATTN | Triton-based attention | General use |
+| ROCM_ATTN | ROCm native attention | MI series GPUs |
+| ROCM_AITER_FA | AITER Flash Attention | Performance |
+| ROCM_AITER_UNIFIED_ATTN | Unified AITER attention | Strix Halo APU |
+| TRITON_MLA | Triton MLA | DeepSeek models |
+| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
+| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |
+
+## Special Flags for APU
+
+### Memory Configuration
+
+Since memory is unified, set these parameters carefully:
+
+```
+vllm serve model \
+    --gpu-memory-utilization 0.85 \
+    --swap-space 8 \
+    --max-num-batched-tokens 4096
+```
+
+- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
+- `swap_space`: Keep moderate (8-16 GB) for overflow
+- `max_model_len`: Match actual context needs
+
+### NUMA Binding
+
+Strix Halo may have multiple NUMA nodes. Use NUMA binding:
+
+```
+vllm serve model \
+    --numa-bind
+```
+
+For explicit binding:
+
+```
+vllm serve model \
+    --numa-bind \
+    --numa-bind-nodes 0 0
+```
+
+### Environment Variables
+
+Set these environment variables for optimal performance:
+
+```
+export ROCM_DISABLE_TCE=0
+export HSA_ENABLE_SDMA=1
+export HSA_FORCE_FINE_GRAIN_PCIE=1
+```
+
+### GPU Memory Limits
+
+Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:
+
+```
+export HSA_OVERRIDE_GFX_VERSION=11.5.1
+```
+
+This may be needed for Strix Halo compatibility.
+
+## Performance Considerations
+
+### Memory Bandwidth
+
+Unified memory has lower bandwidth than dedicated VRAM:
+
+| Memory Type | Bandwidth |
+|-------------|-----------|
+| HBM3 (A100) | 2 TB/s |
+| GDDR6 (RTX 4090) | 1 TB/s |
+| LPDDR5x (Strix Halo) | ~170 GB/s |
+
+Lower bandwidth means slower inference. Plan for lower throughput.
+
+### Cache Effects
+
+Strix Halo has large L3 cache. This helps with memory-bound workloads:
+
+- Large L3 cache reduces memory access latency
+- Prefetching helps with sequential KV cache access
+- Cache eviction may cause stalls during long contexts
+
+### Thermal Throttling
+
+APUs generate more heat than discrete GPUs:
+
+- Monitor GPU temperature with `rocm-smi`
+- Ensure adequate cooling
+- Thermal throttling reduces performance
+- Consider undervolting for sustained workloads
+
+### Concurrency Limits
+
+Unified memory limits concurrent requests:
+
+- Each request uses memory from the shared pool
+- CPU operations also use the shared pool
+- Reduce `max_num_seqs` to avoid memory pressure
+- Monitor memory usage with `rocm-smi`
+
+## Known Limitations
+
+### Feature Support
+
+| Feature | Support on ROCm |
+|---------|-----------------|
+| Tensor Parallelism | Yes |
+| FP8 Quantization | Depends on GPU |
+| CUDA Graphs | Yes |
+| Async Output | No |
+| Multi-step Decoding | No |
+| Speculative Decoding | Yes |
+| Prefix Caching | Yes |
+| Chunked Prefill | Yes |
+
+### Context Length
+
+Some models on ROCm have context length limited to 4096 tokens. Check model documentation.
+
+### FP8 Support
+
+FP8 quantization depends on GPU architecture:
+
+- MI300 series supports native FP8
+- Radeon RX 7900 supports FP8 via software
+- Strix Halo may support FP8 via software emulation
+
+## Example Commands
+
+### Basic APU Deployment
+
+```
+vllm serve Qwen/Qwen3-30B-A3B \
+    --attention-backend ROCM_AITER_UNIFIED_ATTN \
+    --gpu-memory-utilization 0.85 \
+    --swap-space 8 \
+    --max-num-batched-tokens 4096 \
+    --max-num-seqs 64
+```
+
+### With FP8 Model
+
+```
+vllm serve Qwen/Qwen3-30B-A3B-FP8 \
+    --attention-backend ROCM_AITER_UNIFIED_ATTN \
+    --gpu-memory-utilization 0.85 \
+    --kv-cache-dtype fp8 \
+    --swap-space 8
+```
+
+### With Long Context
+
+```
+vllm serve Qwen/Qwen3-30B-A3B \
+    --attention-backend ROCM_AITER_UNIFIED_ATTN \
+    --gpu-memory-utilization 0.8 \
+    --swap-space 16 \
+    --max-model-len 16384 \
+    --rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
+```
+
+## Monitoring
+
+Check GPU status:
+
+```
+rocm-smi
+```
+
+Check ROCm device info:
+
+```
+rocminfo
+```
+
+Check memory usage:
+
+```
+free -h
+```
+
+Monitor temperature:
+
+```
+rocm-smi --showtemperature
+```
+
+## See Also
+
+- [[VLLM Overview]]
+- [[VLLM Index]]
+- [[RAM CPU Offloading in VLLM]]
+- [[Hybrid CPU GPU Processing Dankserver]]
+- [[Running Qwen3.6 27B FP8 on Dankclaw]]