feat: initialize vLLM scripts knowledge base
- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism - VLLM Index: quick reference with commands, endpoints, parameters - Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04 - Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1 - RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption - Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration - Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup - Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide - Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
This commit is contained in:
Frank committed
commit
39db415348
11 files changed
+1577
No files matched your search
@@ -0,0 +1,222 @@
|
||||
# Running Qwen3.6 27B FP8 on Dankclaw
|
||||
|
||||
## Model Information
|
||||
|
||||
Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| HuggingFace ID | `Qwen/Qwen3.6-27B-FP8` |
|
||||
| Base Model | `Qwen/Qwen3.6-27B` |
|
||||
| Architecture | Qwen3_5ForConditionalGeneration |
|
||||
| Total Parameters | 27B |
|
||||
| Hidden Size | 5120 |
|
||||
| Layers | 64 (16 blocks of 3 DeltaNet + 1 Attention) |
|
||||
| Attention Heads | 24 Q, 4 KV (head dim 256) |
|
||||
| DeltaNet Heads | 48 V, 16 QK (head dim 128) |
|
||||
| Intermediate Dim | 17,408 |
|
||||
| Vocab Size | 248,320 (padded) |
|
||||
| Context Length | 262,144 (natively extensible to 1,010,000) |
|
||||
| Quantization | FP8 (block size 128) |
|
||||
| Vision Encoder | Yes (image-text-to-text) |
|
||||
|
||||
## Hardware: Dankclaw
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
|
||||
| GPU | Integrated Radeon 8060S (unified memory) |
|
||||
| Unified RAM | 61 GB (shared CPU + GPU) |
|
||||
| Swap | 8 GB |
|
||||
| ROCm/HIP | 7.1 |
|
||||
| Python | 3.14.4 |
|
||||
|
||||
## Memory Calculation
|
||||
|
||||
| Component | BF16 | FP8 |
|
||||
|-----------|------|-----|
|
||||
| Model weights (27B) | 54 GB | 13.5 GB |
|
||||
| Vision encoder | ~1 GB | ~0.5 GB |
|
||||
| **Total weights** | ~55 GB | ~14 GB |
|
||||
| KV cache (64K ctx, 1 seq) | ~1 GB | ~0.5 GB |
|
||||
| Overhead | ~2 GB | ~2 GB |
|
||||
| **Total (approximate)** | ~58 GB | ~16.5 GB |
|
||||
|
||||
The FP8 variant requires approximately 16-17 GB. This fits within the 61 GB unified memory with comfortable headroom for KV cache.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install vLLM for ROCm. Version 0.19.0 or later is required for Qwen3.6:
|
||||
|
||||
```
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
Python 3.14.4 is installed on Dankclaw. vLLM ROCm wheels target Python 3.12. Create a virtual environment with Python 3.12 if available. Verify that Python 3.12 is installed:
|
||||
|
||||
```
|
||||
python3.12 --version
|
||||
```
|
||||
|
||||
If Python 3.12 is not available, install it or use the system Python with `--no-build-isolation`.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Set these for ROCm on Strix Halo:
|
||||
|
||||
```
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export HF_HOME=/home/dank/.cache/huggingface
|
||||
```
|
||||
|
||||
## vLLM Serve Commands
|
||||
|
||||
### Text-Only (Recommended for Memory Savings)
|
||||
|
||||
Skip the vision encoder to reduce memory:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 32 \
|
||||
--reasoning-parser qwen3 \
|
||||
--language-model-only \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### With Vision Encoder
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 32 \
|
||||
--reasoning-parser qwen3 \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### Parameters Explained
|
||||
|
||||
| Parameter | Value | Reason |
|
||||
|-----------|-------|--------|
|
||||
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
|
||||
| --gpu-memory-utilization 0.85 | 85% of unified memory | Leaves 15% for CPU/OS |
|
||||
| --swap-space 8 | 8 GB CPU swap | Matches system swap size |
|
||||
| --max-model-len 131072 | 128K context | Full 262K is large but possible with 61 GB |
|
||||
| --language-model-only | Skip vision | Saves ~1 GB for KV cache |
|
||||
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode |
|
||||
|
||||
## Sampling Parameters
|
||||
|
||||
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|
||||
|------|-------------|-------|-------|-------|-----------------|
|
||||
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
|
||||
|
||||
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
|
||||
|
||||
## API Examples
|
||||
|
||||
### Chat with Thinking
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-27B-FP8",
|
||||
"messages": [{"role": "user", "content": "Solve: 2+2"}],
|
||||
"max_tokens": 32768,
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20
|
||||
}'
|
||||
```
|
||||
|
||||
### Non-Thinking Mode
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-27B-FP8",
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"max_tokens": 2048,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.8,
|
||||
"top_k": 20,
|
||||
"presence_penalty": 1.5,
|
||||
"extra_body": {
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
## Ultra-Long Context (YaRN)
|
||||
|
||||
Extend context beyond 262,144 tokens:
|
||||
|
||||
```
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
|
||||
--max-model-len 524288
|
||||
```
|
||||
|
||||
## ROCm Notes
|
||||
|
||||
- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
|
||||
- FP8 may use software emulation on RDNA architecture. Verify with a BF16 test first
|
||||
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
|
||||
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Out of Memory
|
||||
|
||||
1. Use `--language-model-only` to skip vision encoder
|
||||
2. Reduce `--gpu-memory-utilization` to 0.75
|
||||
3. Reduce `--max-model-len` to 65536
|
||||
4. Reduce `--max-num-batched-tokens` to 1024
|
||||
|
||||
### FP8 Errors
|
||||
|
||||
1. Try BF16 first: `vllm serve Qwen/Qwen3.6-27B --dtype bfloat16`
|
||||
2. Verify ROCm version supports FP8
|
||||
3. Set `HSA_OVERRIDE_GFX_VERSION=11.5.1`
|
||||
4. Use `--kv-cache-dtype auto` instead of fp8
|
||||
|
||||
### Slow Inference
|
||||
|
||||
1. Check GPU utilization with `rocm-smi`
|
||||
2. Verify attention backend is optimal
|
||||
3. Increase `--max-num-batched-tokens` if memory allows
|
||||
4. Ensure CPU cores are not throttled
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[Dankclaw System Specs]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
Reference in new issue
Block a user