serverAppConfigs/docs/Running Qwen3.6 27B FP8 on Dankclaw.md

Running Qwen3.6 27B FP8 on Dankclaw

Model Information

Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.

Property	Value
HuggingFace ID	Qwen/Qwen3.6-27B-FP8
Base Model	Qwen/Qwen3.6-27B
Architecture	Qwen3_5ForConditionalGeneration
Total Parameters	27B
Hidden Size	5120
Layers	64 (16 blocks of 3 DeltaNet + 1 Attention)
Attention Heads	24 Q, 4 KV (head dim 256)
DeltaNet Heads	48 V, 16 QK (head dim 128)
Intermediate Dim	17,408
Vocab Size	248,320 (padded)
Context Length	262,144 (natively extensible to 1,010,000)
Quantization	FP8 (block size 128)
Vision Encoder	Yes (image-text-to-text)

Hardware: Dankclaw

Property	Value
CPU	AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo)
GPU	Integrated Radeon 8060S (unified memory)
Unified RAM	61 GB (shared CPU + GPU)
Swap	8 GB
ROCm/HIP	7.1
Python	3.14.4

Memory Calculation

Component	BF16	FP8
Model weights (27B)	54 GB	13.5 GB
Vision encoder	~1 GB	~0.5 GB
Total weights	~55 GB	~14 GB
KV cache (64K ctx, 1 seq)	~1 GB	~0.5 GB
Overhead	~2 GB	~2 GB
Total (approximate)	~58 GB	~16.5 GB

The FP8 variant requires approximately 16-17 GB. This fits within the 61 GB unified memory with comfortable headroom for KV cache.

Prerequisites

Install vLLM for ROCm. Version 0.19.0 or later is required for Qwen3.6:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Python 3.14.4 is installed on Dankclaw. vLLM ROCm wheels target Python 3.12. Create a virtual environment with Python 3.12 if available. Verify that Python 3.12 is installed:

python3.12 --version

If Python 3.12 is not available, install it or use the system Python with --no-build-isolation.

Environment Variables

Set these for ROCm on Strix Halo:

export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface

vLLM Serve Commands

Text-Only (Recommended for Memory Savings)

Skip the vision encoder to reduce memory:

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --language-model-only \
    --dtype auto

With Vision Encoder

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto

Parameters Explained

Parameter	Value	Reason
--kv-cache-dtype fp8	FP8 KV cache	Cuts KV cache memory by 50%
--gpu-memory-utilization 0.85	85% of unified memory	Leaves 15% for CPU/OS
--swap-space 8	8 GB CPU swap	Matches system swap size
--max-model-len 131072	128K context	Full 262K is large but possible with 61 GB
--language-model-only	Skip vision	Saves ~1 GB for KV cache
--reasoning-parser qwen3	Qwen3 parser	Required for thinking mode

Sampling Parameters

Mode	Temperature	Top P	Top K	Min P	Presence Penalty
Thinking (general)	1.0	0.95	20	0.0	0.0
Thinking (coding)	0.6	0.95	20	0.0	0.0
Non-thinking	0.7	0.80	20	0.0	1.5

Set max_tokens to 32768 for general use. Set to 81920 for complex reasoning.

API Examples

Chat with Thinking

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'

Non-Thinking Mode

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'

Tool Calling

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288

ROCm Notes

• Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
• FP8 may use software emulation on RDNA architecture. Verify with a BF16 test first
• Attention backends: try ROCM_AITER_UNIFIED_ATTN if the default backend underperforms
• Context length on ROCm may be limited for some models. Start with 131072 and increase gradually

Troubleshooting

Out of Memory

1. Use --language-model-only to skip vision encoder
2. Reduce --gpu-memory-utilization to 0.75
3. Reduce --max-model-len to 65536
4. Reduce --max-num-batched-tokens to 1024

FP8 Errors

1. Try BF16 first: vllm serve Qwen/Qwen3.6-27B --dtype bfloat16
2. Verify ROCm version supports FP8
3. Set HSA_OVERRIDE_GFX_VERSION=11.5.1
4. Use --kv-cache-dtype auto instead of fp8

Slow Inference

1. Check GPU utilization with rocm-smi
2. Verify attention backend is optimal
3. Increase --max-num-batched-tokens if memory allows
4. Ensure CPU cores are not throttled

See Also

• VLLM Overview
• Dankclaw System Specs
• Shared Memory Architecture Dankclaw
• RAM CPU Offloading in VLLM
• Running Qwen3.6 35B A3B FP8 on Dankserver