dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initialize vLLM scripts knowledge base

Browse Source 
- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism
- VLLM Index: quick reference with commands, endpoints, parameters
- Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04
- Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1
- RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption
- Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration
- Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup
- Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide
- Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
This commit is contained in:
Frank
2026-08-05 23:49:02 -07:00
commit 39db415348
11 changed files with 1577 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -0,0 +1,5 @@
.pi-subagents/
.AGENTTRASH/
__pycache__/
*.pyc
.venv/
README.md
+26
View File
@@ -0,0 +1,26 @@
# vLLM Scripts
Scripts for running large language models via vLLM on two local machines.
## Targets
| Host | User | Codename | Architecture |
|------|------|----------|-------------|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
## Models
| Model | Target | Format |
|-------|--------|--------|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
## Structure
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
- `scripts/` — Executable vLLM launch scripts (coming soon)
## Knowledge Base
See [[docs/README.md]] for the project knowledge base index.
docs/Dankclaw System Specs.md
+75
View File
@@ -0,0 +1,75 @@
# Dankclaw System Specs
## Server Overview
Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same 61 GB RAM pool.
## Hardware Specifications
### CPU
| Property | Value |
|----------|-------|
| Model | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
| Architecture | x86_64 |
| Cores | 16 (1 socket, 1 thread per core) |
| Max MHz | 5187.50 |
| Min MHz | 625.00 |
| NUMA Nodes | 1 (node 0, 62909 MB) |
### GPU (Integrated)
| Property | Value |
|----------|-------|
| Model | AMD Radeon 8060S (Strix Halo) |
| PCIe ID | c4:00.0 |
| Architecture | Strix/Strix Halo |
| Memory | Unified with system RAM (no separate VRAM) |
### ROCm
| Property | Value |
|----------|-------|
| ROCk Module | Loaded |
| Runtime Version | 1.1 |
| HIP Version | 7.1.52801 |
| Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
### Memory
| Property | Value |
|----------|-------|
| System RAM | 61 GiB (shared CPU + GPU) |
| Swap | 8.0 GiB |
| /dev/shm | 31 GiB |
### Storage
| Filesystem | Size | Used | Avail | Use% |
|------------|------|------|-------|------|
| / (nvme0n1p2) | 1.9T | 333G | 1.5T | 19% |
### OS and Software
| Property | Value |
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-28-generic |
| Python | 3.14.4 |
## VLLM Relevance
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- 61 GB shared memory allows large models without VRAM constraints.
- Only 8 GB swap. Set `--swap-space` conservatively (4-8 GB).
- ROCm and HIP 7.1 are installed. vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Verify compatibility.
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151/1150). vLLM lists this as supported.
- 1.5 TB free disk space is ample for model storage.
## See Also
- [[VLLM Overview]]
- [[Shared Memory Architecture Dankclaw]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
docs/Dankserver System Specs.md
+68
View File
@@ -0,0 +1,68 @@
# Dankserver System Specs
## Server Overview
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.04 with an RTX 3090 and 62 GB of system RAM.
## Hardware Specifications
### CPU
| Property | Value |
|----------|-------|
| Architecture | x86_64 |
| Vendor | AuthenticAMD |
| Cores | 8 (1 socket) |
| NUMA Nodes | 1 (node 0, 63503 MB) |
### GPU
| Property | Value |
|----------|-------|
| Model | NVIDIA GeForce RTX 3090 |
| VRAM | 24576 MiB (24 GB) |
| Driver | 595.84 |
| CUDA Version | 13.2 |
| CUDA Toolkit | 12.0 (nvcc) |
### Memory
| Property | Value |
|----------|-------|
| System RAM | 62 GiB |
| Swap | 127 GiB |
| /dev/shm | 32 GiB |
### Storage
| Filesystem | Size | Used | Avail | Use% |
|------------|------|------|-------|------|
| / (nvme0n1p2) | 915G | 849G | 20G | 98% |
| /mnt/entropy (dm-0) | 91T | 86T | 977G | 99% |
### OS and Software
| Property | Value |
|----------|-------|
| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
| Kernel | 7.0.0-28-generic |
| Python | 3.12.3 |
### Active Services
A llama-server process occupies approximately 20 GB of the RTX 3090 VRAM.
## VLLM Relevance
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
- 62 GB system RAM provides ample swap space for KV cache offloading.
- 127 GB swap supports large `--swap-space` values for CPU KV cache paging.
- Root disk has only 20 GB free. Store models on /mnt/entropy.
- CUDA 12.0 and Python 3.12.3 are compatible with vLLM.
## See Also
- [[VLLM Overview]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
docs/Hybrid CPU GPU Processing Dankserver.md
+213
View File
@@ -0,0 +1,213 @@
# Hybrid CPU GPU Processing Dankserver
## Overview
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. VLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
## Architecture
### Model Weights
Model weights stay on GPU. VLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
### KV Cache
KV cache moves between GPU and CPU:
1. Active KV cache pages stay in GPU VRAM
2. Inactive pages swap to CPU RAM
3. Swapped pages write to disk swap space if RAM is full
4. Pages reload to GPU when needed
### Compute Distribution
- GPU handles all matrix multiplication and attention computation
- CPU handles tokenization, detokenization, and input processing
- CPU handles KV cache swap operations
- GPU worker processes run on CPU cores via multiprocessing
## Optimal Settings for Dankserver
### GPU Memory Utilization
Set `--gpu-memory-utilization` based on your GPU:
| GPU | Recommended Value |
|-----|-------------------|
| 24 GB (RTX 3090/4090) | 0.7-0.8 |
| 40 GB (A100) | 0.8-0.9 |
| 80 GB (A100 80GB) | 0.85-0.95 |
| 80 GB (H100) | 0.9-0.95 |
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
### Swap Space
Set `--swap-space` based on system RAM:
| System RAM | Recommended Swap |
|------------|------------------|
| 32 GB | 8-16 GB |
| 64 GB | 16-32 GB |
| 128 GB | 32-64 GB |
| 256 GB | 64-128 GB |
More swap space allows longer contexts and more concurrent requests.
### Tensor Parallelism
Use tensor parallelism for large models:
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.85 \
--swap-space 16
```
### KV Cache Data Type
Use FP8 KV cache for Ampere and newer GPUs:
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16
```
FP8 KV cache reduces memory usage by 50% compared to BF16.
## NUMA Optimization
Dankserver may have multiple NUMA nodes. Optimize with:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind
```
This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
For explicit NUMA binding:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind \
--numa-bind-nodes 0 0 1 1
```
## CPU Resource Requirements
VLLM needs CPU cores for:
- 1 API server process
- 1 engine core process
- N GPU worker processes (one per GPU)
Minimum physical cores = 2 + N (where N is GPU count).
For Dankserver with 1 GPU:
- Minimum 3 physical CPU cores
- Recommended 8+ physical CPU cores
## Performance Tuning
### Chunked Prefill
Chunked prefill is enabled by default. Tune with:
```
vllm serve model \
--max-num-batched-tokens 8192
```
Higher values improve throughput. Lower values improve latency.
### CUDA Graphs
CUDA graphs improve throughput. Disable for faster startup:
```
vllm serve model \
--enforce-eager
```
### Compilation Cache
Reuse torch.compile artifacts:
```
export VLLM_CACHE_ROOT=~/.cache/vllm
vllm serve model
```
Copy cache between boots for faster startup.
### Skip Memory Profiling
Skip memory profiling on repeated boots:
```
vllm serve model \
--kv-cache-memory <value_from_first_boot_log>
```
## Monitoring
Check GPU memory:
```
nvidia-smi
```
Check for preemption warnings in VLLM logs.
Check CPU utilization:
```
htop
```
## Example Commands
### Single GPU, Medium Model
```
vllm serve Qwen/Qwen3-30B-A3B \
--gpu-memory-utilization 0.85 \
--swap-space 16 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Single GPU, FP8 Model
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16 \
--max-num-batched-tokens 8192
```
### Multi-GPU, Large Model
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--swap-space 32 \
--max-model-len 8192
```
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
docs/RAM CPU Offloading in VLLM.md
+179
View File
@@ -0,0 +1,179 @@
# RAM CPU Offloading in VLLM
## Overview
VLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. VLLM does not offload model weights to CPU. Instead, VLLM moves KV cache data between GPU and CPU.
## How CPU Offloading Works
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
The KV cache swap mechanism works as follows:
1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space
## Key Parameters
### gpu_memory_utilization
This parameter controls the fraction of GPU memory used for KV cache.
| Value | Description |
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.9 | Uses 90% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space
This parameter sets the CPU swap space in GiB.
| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
| 64 | 64 GiB swap space |
The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.
### max_model_len
This parameter sets the maximum context length.
| Value | Description |
|-------|-------------|
| auto | VLLM calculates optimal length |
| 2048 | 2048 tokens |
| 4096 | 4096 tokens |
| 8192 | 8192 tokens |
| 32768 | 32768 tokens |
Smaller values reduce memory usage but limit context length.
### max_num_batched_tokens
This parameter sets the maximum number of tokens per batch.
| Value | Description |
|-------|-------------|
| 2048 | Default for online serving |
| 4096 | Moderate batch size |
| 8192 | Large batch size |
| 16384 | Very large batch size |
### max_num_seqs
This parameter sets the maximum number of sequences per batch.
| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Default for offline inference |
| 512 | Large batch |
### kv_cache_dtype
This parameter sets the KV cache data type.
| Value | Description |
|-------|-------------|
| auto | Uses model default dtype |
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |
## Memory Configuration Examples
### Small Model on Single GPU
```
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
--gpu-memory-utilization 0.9 \
--swap-space 4 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Large Model with CPU Offload
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
--max-model-len 8192
```
### Memory-Constrained GPU
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.7 \
--swap-space 32 \
--max-model-len 4096 \
--max-num-batched-tokens 2048
```
## Preemption Modes
VLLM uses two preemption modes when GPU memory is insufficient:
### Recompute Mode (Default in V1)
VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.
### Swap Mode
VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.
## Monitoring
Watch for preemption warnings:
```
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
```
If you see frequent preemptions:
1. Increase `gpu_memory_utilization`
2. Increase `swap_space`
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage
## Memory Conservation Tips
- Use quantization (FP8, INT8) to reduce model size
- Reduce `max_model_len` to match actual needs
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Use `--kv-cache-memory` to skip memory profiling
- Limit multi-modal inputs with `limit_mm_per_prompt`
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| VLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| VLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| VLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| VLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
docs/Running Qwen3.6 27B FP8 on Dankclaw.md
+222
View File
@@ -0,0 +1,222 @@
# Running Qwen3.6 27B FP8 on Dankclaw
## Model Information
Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.
| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-27B-FP8` |
| Base Model | `Qwen/Qwen3.6-27B` |
| Architecture | Qwen3_5ForConditionalGeneration |
| Total Parameters | 27B |
| Hidden Size | 5120 |
| Layers | 64 (16 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 24 Q, 4 KV (head dim 256) |
| DeltaNet Heads | 48 V, 16 QK (head dim 128) |
| Intermediate Dim | 17,408 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |
## Hardware: Dankclaw
| Property | Value |
|----------|-------|
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
| GPU | Integrated Radeon 8060S (unified memory) |
| Unified RAM | 61 GB (shared CPU + GPU) |
| Swap | 8 GB |
| ROCm/HIP | 7.1 |
| Python | 3.14.4 |
## Memory Calculation
| Component | BF16 | FP8 |
|-----------|------|-----|
| Model weights (27B) | 54 GB | 13.5 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~55 GB | ~14 GB |
| KV cache (64K ctx, 1 seq) | ~1 GB | ~0.5 GB |
| Overhead | ~2 GB | ~2 GB |
| **Total (approximate)** | ~58 GB | ~16.5 GB |
The FP8 variant requires approximately 16-17 GB. This fits within the 61 GB unified memory with comfortable headroom for KV cache.
## Prerequisites
Install vLLM for ROCm. Version 0.19.0 or later is required for Qwen3.6:
```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
Python 3.14.4 is installed on Dankclaw. vLLM ROCm wheels target Python 3.12. Create a virtual environment with Python 3.12 if available. Verify that Python 3.12 is installed:
```
python3.12 --version
```
If Python 3.12 is not available, install it or use the system Python with `--no-build-isolation`.
## Environment Variables
Set these for ROCm on Strix Halo:
```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface
```
## vLLM Serve Commands
### Text-Only (Recommended for Memory Savings)
Skip the vision encoder to reduce memory:
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--language-model-only \
--dtype auto
```
### With Vision Encoder
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--dtype auto
```
### Parameters Explained
| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.85 | 85% of unified memory | Leaves 15% for CPU/OS |
| --swap-space 8 | 8 GB CPU swap | Matches system swap size |
| --max-model-len 131072 | 128K context | Full 262K is large but possible with 61 GB |
| --language-model-only | Skip vision | Saves ~1 GB for KV cache |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode |
## Sampling Parameters
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
## API Examples
### Chat with Thinking
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-27B-FP8",
"messages": [{"role": "user", "content": "Solve: 2+2"}],
"max_tokens": 32768,
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20
}'
```
### Non-Thinking Mode
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-27B-FP8",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 2048,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"presence_penalty": 1.5,
"extra_body": {
"chat_template_kwargs": {"enable_thinking": false}
}
}'
```
### Tool Calling
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
## Ultra-Long Context (YaRN)
Extend context beyond 262,144 tokens:
```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
--max-model-len 524288
```
## ROCm Notes
- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
- FP8 may use software emulation on RDNA architecture. Verify with a BF16 test first
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually
## Troubleshooting
### Out of Memory
1. Use `--language-model-only` to skip vision encoder
2. Reduce `--gpu-memory-utilization` to 0.75
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 1024
### FP8 Errors
1. Try BF16 first: `vllm serve Qwen/Qwen3.6-27B --dtype bfloat16`
2. Verify ROCm version supports FP8
3. Set `HSA_OVERRIDE_GFX_VERSION=11.5.1`
4. Use `--kv-cache-dtype auto` instead of fp8
### Slow Inference
1. Check GPU utilization with `rocm-smi`
2. Verify attention backend is optimal
3. Increase `--max-num-batched-tokens` if memory allows
4. Ensure CPU cores are not throttled
## See Also
- [[VLLM Overview]]
- [[Dankclaw System Specs]]
- [[Shared Memory Architecture Dankclaw]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md
+195
View File
@@ -0,0 +1,195 @@
# Running Qwen3.6 35B A3B FP8 on Dankserver
## Model Information
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-35B-A3B-FP8` |
| Base Model | `Qwen/Qwen3.6-35B-A3B` |
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Total Parameters | 35B |
| Activated Parameters | 3B |
| Experts | 256 (8 routed + 1 shared per token) |
| Layers | 40 (10 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 16 Q, 2 KV (head dim 256) |
| DeltaNet Heads | 32 V, 16 QK (head dim 128) |
| Expert Intermediate Dim | 512 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |
## Hardware: Dankserver
| Property | Value |
|----------|-------|
| GPU | NVIDIA GeForce RTX 3090 |
| VRAM | 24 GB |
| System RAM | 62 GB |
| Swap | 127 GB |
## Memory Calculation
| Component | BF16 | FP8 |
|-----------|------|-----|
| Activated weights (3B) | 6 GB | 1.5 GB |
| Inactive experts (32B) | 64 GB | 16 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~71 GB | ~18 GB |
The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.
## Prerequisites
Install vLLM. Version 0.19.0 or later is required for Qwen3.6:
```
uv pip install vllm --torch-backend=auto
```
## vLLM Serve Commands
### Text-Only (Recommended for Memory Savings)
Skip the vision encoder to free VRAM for KV cache:
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.75 \
--swap-space 32 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--dtype auto
```
### With Vision Encoder
If you need multimodal input, expect higher VRAM usage:
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.7 \
--swap-space 32 \
--max-model-len 131072 \
--max-num-batched-tokens 1024 \
--max-num-seqs 16 \
--reasoning-parser qwen3 \
--dtype auto
```
### Parameters Explained
| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
| --swap-space 32 | 32 GB CPU swap | Dankserver has 62 GB RAM |
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |
### Model Storage
Root disk has only 20 GB free. Store models on /mnt/entropy:
```
export HF_HOME=/mnt/entropy/huggingface
```
### Active Service Conflict
A llama-server process occupies approximately 20 GB of VRAM. Stop it before starting vLLM:
```
pkill llama-server
nvidia-smi # verify GPU is free
```
## Sampling Parameters
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
## API Examples
### Chat with Thinking
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role": "user", "content": "Solve: 2+2"}],
"max_tokens": 32768,
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20
}'
```
### Non-Thinking Mode
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 2048,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"presence_penalty": 1.5,
"extra_body": {
"chat_template_kwargs": {"enable_thinking": false}
}
}'
```
## Ultra-Long Context (YaRN)
Extend context beyond 262,144 tokens. Use this environment variable and override:
```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
--max-model-len 524288
```
## Troubleshooting
### Out of Memory
1. Kill llama-server to free 20 GB VRAM
2. Use `--language-model-only` to skip vision encoder
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 512
5. Reduce `--max-num-seqs` to 8
### Slow Inference
1. Verify GPU is exclusive to vLLM
2. Check temperature with `nvidia-smi`
3. Ensure `--kv-cache-dtype fp8` is set
4. Increase `--max-num-batched-tokens` if memory allows
## See Also
- [[VLLM Overview]]
- [[Dankserver System Specs]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
docs/Shared Memory Architecture Dankclaw.md
+265
View File
@@ -0,0 +1,265 @@
# Shared Memory Architecture Dankclaw
## Overview
Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.
## Unified Memory Architecture
### How Unified Memory Works
In unified memory:
1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to VLLM
5. Memory pages move between CPU and GPU transparently
### Implications for VLLM
- VLLM sees the full memory as GPU memory
- No explicit swap_space parameter needed for memory extension
- The GPU driver handles memory paging automatically
- Performance depends on memory bandwidth and latency
- Shared memory means CPU and GPU compete for the same pool
## ROCm Configuration
### ROCm Version
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw should have ROCm 7.0 or later.
Check ROCm version:
```
rocminfo
```
### GPU Architecture
Strix Halo uses the gfx115x architecture. Check with:
```
rocminfo | grep gfx
```
### Installation
Install VLLM for ROCm with Python 3.12:
```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
### Attention Backend
Choose the best attention backend for Strix Halo:
```
vllm serve model \
--attention-backend ROCM_AITER_UNIFIED_ATTN
```
Available backends for ROCm:
| Backend | Description | Best For |
|---------|-------------|----------|
| TRITON_ATTN | Triton-based attention | General use |
| ROCM_ATTN | ROCm native attention | MI series GPUs |
| ROCM_AITER_FA | AITER Flash Attention | Performance |
| ROCM_AITER_UNIFIED_ATTN | Unified AITER attention | Strix Halo APU |
| TRITON_MLA | Triton MLA | DeepSeek models |
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |
## Special Flags for APU
### Memory Configuration
Since memory is unified, set these parameters carefully:
```
vllm serve model \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-num-batched-tokens 4096
```
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
- `swap_space`: Keep moderate (8-16 GB) for overflow
- `max_model_len`: Match actual context needs
### NUMA Binding
Strix Halo may have multiple NUMA nodes. Use NUMA binding:
```
vllm serve model \
--numa-bind
```
For explicit binding:
```
vllm serve model \
--numa-bind \
--numa-bind-nodes 0 0
```
### Environment Variables
Set these environment variables for optimal performance:
```
export ROCM_DISABLE_TCE=0
export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1
```
### GPU Memory Limits
Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:
```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
```
This may be needed for Strix Halo compatibility.
## Performance Considerations
### Memory Bandwidth
Unified memory has lower bandwidth than dedicated VRAM:
| Memory Type | Bandwidth |
|-------------|-----------|
| HBM3 (A100) | 2 TB/s |
| GDDR6 (RTX 4090) | 1 TB/s |
| LPDDR5x (Strix Halo) | ~170 GB/s |
Lower bandwidth means slower inference. Plan for lower throughput.
### Cache Effects
Strix Halo has large L3 cache. This helps with memory-bound workloads:
- Large L3 cache reduces memory access latency
- Prefetching helps with sequential KV cache access
- Cache eviction may cause stalls during long contexts
### Thermal Throttling
APUs generate more heat than discrete GPUs:
- Monitor GPU temperature with `rocm-smi`
- Ensure adequate cooling
- Thermal throttling reduces performance
- Consider undervolting for sustained workloads
### Concurrency Limits
Unified memory limits concurrent requests:
- Each request uses memory from the shared pool
- CPU operations also use the shared pool
- Reduce `max_num_seqs` to avoid memory pressure
- Monitor memory usage with `rocm-smi`
## Known Limitations
### Feature Support
| Feature | Support on ROCm |
|---------|-----------------|
| Tensor Parallelism | Yes |
| FP8 Quantization | Depends on GPU |
| CUDA Graphs | Yes |
| Async Output | No |
| Multi-step Decoding | No |
| Speculative Decoding | Yes |
| Prefix Caching | Yes |
| Chunked Prefill | Yes |
### Context Length
Some models on ROCm have context length limited to 4096 tokens. Check model documentation.
### FP8 Support
FP8 quantization depends on GPU architecture:
- MI300 series supports native FP8
- Radeon RX 7900 supports FP8 via software
- Strix Halo may support FP8 via software emulation
## Example Commands
### Basic APU Deployment
```
vllm serve Qwen/Qwen3-30B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-num-batched-tokens 4096 \
--max-num-seqs 64
```
### With FP8 Model
```
vllm serve Qwen/Qwen3-30B-A3B-FP8 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--kv-cache-dtype fp8 \
--swap-space 8
```
### With Long Context
```
vllm serve Qwen/Qwen3-30B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
--max-model-len 16384 \
--rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
```
## Monitoring
Check GPU status:
```
rocm-smi
```
Check ROCm device info:
```
rocminfo
```
Check memory usage:
```
free -h
```
Monitor temperature:
```
rocm-smi --showtemperature
```
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
docs/VLLM Index.md
+93
View File
@@ -0,0 +1,93 @@
# VLLM Index
## Documentation Overview
This index contains all documentation for VLLM deployment on Dankserver and Dankclaw.
## Hardware Guides
- [[Dankserver System Specs]] -- NVIDIA GPU server specifications
- [[Dankclaw System Specs]] -- AMD GPU server specifications
## VLLM Architecture
- [[VLLM Overview]] -- Complete VLLM documentation including installation, CLI, API, and key arguments
- [[RAM CPU Offloading in VLLM]] -- CPU RAM offloading mechanisms, swap space, and memory configuration
- [[Hybrid CPU GPU Processing Dankserver]] -- NVIDIA GPU with system RAM hybrid processing
- [[Shared Memory Architecture Dankclaw]] -- AMD ROCm unified memory and APU configuration
## Model Deployment Guides
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] -- Qwen3-30B-A3B FP8 deployment on NVIDIA GPU
- [[Running Qwen3.6 27B FP8 on Dankclaw]] -- Qwen3-32B FP8 deployment on AMD GPU
## Quick Reference
### Installation Commands
```
# NVIDIA GPU
uv pip install vllm --torch-backend=auto
# AMD ROCm
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
# CPU
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
```
### Server Start Commands
```
# Basic server
vllm serve Qwen/Qwen2.5-1.5B-Instruct
# With tensor parallelism
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
# With quantization
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8
# With GPU memory control
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space 8
```
### Common API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| /v1/models | GET | List available models |
| /v1/completions | POST | Text completion |
| /v1/chat/completions | POST | Chat completion |
| /health | GET | Health check |
| /metrics | GET | Prometheus metrics |
### Key Parameters
| Parameter | Description | Typical Value |
|-----------|-------------|---------------|
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
| gpu_memory_utilization | Fraction of GPU memory for KV cache | 0.7-0.95 |
| swap_space | CPU swap space in GiB | 4-32 |
| max_model_len | Maximum context length | auto |
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
| max_num_seqs | Max sequences per batch | 128-256 |
### Quantization Methods
| Method | GPU Support | Description |
|--------|-------------|-------------|
| FP8 | Hopper, Ada, Ampere | Native FP8 inference |
| INT8 | Most GPUs | Weight-only INT8 |
| GPTQ | Most GPUs | Post-training GPTQ |
| AWQ | Turing+ | Activation-aware weight quantization |
| GGUF | CPU, GPU | General GPU unification format |
## See Also
- [[VLLM Overview]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
docs/VLLM Overview.md
+236
View File
@@ -0,0 +1,236 @@
# VLLM Overview
## What Is VLLM
VLLM is a fast and easy library for large language model inference and serving. You can use VLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. VLLM supports over 200 model architectures from Hugging Face.
VLLM is fast with these features:
- PagedAttention for efficient key and value memory management
- Continuous batching of incoming requests
- Chunked prefill and prefix caching
- FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
- Tensor, pipeline, data, expert, and context parallelism
- Speculative decoding with n-gram, suffix, and EAGLE
- Automatic kernel generation with torch.compile
- Disaggregated prefill, decode, and encode
VLLM is flexible with these features:
- Seamless integration with Hugging Face models
- High-throughput serving with parallel sampling and beam search
- Streaming outputs
- Structured outputs with xgrammar or guidance
- Tool calling and reasoning parsers
- OpenAI-compatible API server
- Anthropic Messages API and gRPC support
- Efficient multi-LoRA support
## Installation
You can install VLLM with uv (recommended) or pip:
```
uv pip install vllm
```
Or with pip for NVIDIA GPUs:
```
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
```
For AMD GPUs with ROCm:
```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
For CPU:
```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
```
## Quick Start
### Offline Batched Inference
```python
from vllm import LLM, SamplingParams
llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
prompts = [
"Hello, my name is",
"The capital of France is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
generated_text = output.outputs[0].text
print(generated_text)
```
### Online Serving
Start the OpenAI-compatible API server:
```
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```
Query the server:
```
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'
```
Chat completions:
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
## Key Arguments
### Model Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `model` | required | Model name or path |
| `tokenizer` | same as model | Tokenizer name or path |
| `tokenizer_mode` | auto | Tokenizer mode |
| `trust_remote_code` | false | Trust remote code from Hugging Face |
| `dtype` | auto | Data type (auto, float16, bfloat16, float32) |
| `max_model_len` | auto | Maximum context length |
| `quantization` | null | Quantization method (awq, gptq, fp8, compressed-tensors) |
| `load_format` | auto | Weight loading format |
### GPU Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
| `gpu_memory_utilization` | 0.9 | Fraction of GPU memory for KV cache |
| `swap_space` | 4 | CPU swap space in GiB |
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `max_num_seqs` | 256 | Max sequences per batch |
| `block_size` | 16 | Token block size |
| `num_gpu_blocks_override` | null | Override number of GPU blocks |
| `max_model_len_per_seq` | null | Max length per sequence |
### Optimization Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `disable_log_stats` | false | Disable logging |
| `disable_log_requests` | false | Disable request logging |
| `max_log_len` | null | Max log length |
| `disable_custom_all_reduce` | false | Disable custom all-reduce |
| `tokenizer_pool_size` | 0 | Tokenizer pool size |
| `tokenizer_pool_type` | none | Tokenizer pool type |
| `enable_chunked_prefill` | auto | Enable chunked prefill |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `attention_backend` | auto | Attention backend (flash_attn, flashinfer) |
### Memory Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `gpu_memory_utilization` | 0.9 | Fraction of GPU memory for KV cache |
| `swap_space` | 4 | CPU swap space in GiB |
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
| `max_model_len` | auto | Maximum context length |
| `num_gpu_blocks` | auto | Number of GPU blocks |
| `num_cpu_blocks` | auto | Number of CPU blocks |
### Device Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `device` | auto | Device type (cuda, rocm, xpu, cpu, metal) |
| `distributed_executor_backend` | auto | Distributed executor backend |
| `huggingface_tokenizer_kwargs` | null | Hugging Face tokenizer kwargs |
## Architecture
VLLM uses a multi-process architecture:
- API server process handles HTTP requests, tokenization, and input processing
- Engine core process runs the scheduler and coordinates GPU workers
- One GPU worker process per GPU executes model forward passes
VLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.
VLLM supports multiple attention backends:
- FlashAttention for NVIDIA GPUs
- FlashInfer for NVIDIA GPUs
- Triton attention for AMD ROCm
- ROCm attention for AMD ROCm
## Supported Models
VLLM supports over 200 model architectures including:
- Decoder-only LLMs (Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
- Hybrid attention and state-space models (Mamba, Qwen3.5)
- Multi-modal models (LLaVA, Qwen-VL, Pixtral)
- Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
- Reward and classification models (Qwen-Math)
## Quantization
VLLM supports these quantization methods:
- FP8 for Hopper and Ada GPUs
- MXFP8/MXFP4 for Hopper GPUs
- NVFP4 for Hopper GPUs
- INT8 for most GPUs
- INT4 for most GPUs
- GPTQ for most GPUs
- AWQ for Turing and newer GPUs
- GGUF for CPU and GPU
- Compressed-tensors for custom quantization
- ModelOpt for NVIDIA Model Optimizer
- TorchAO for PyTorch AO
## Parallelism
VLLM supports multiple parallelism strategies:
- Tensor Parallelism (TP) shards model parameters across GPUs
- Pipeline Parallelism (PP) distributes model layers across GPUs
- Data Parallelism (DP) replicates the model across GPU sets
- Expert Parallelism (EP) distributes expert networks in MoE models
- Context Parallelism (CP) splits context across GPUs
## See Also
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]