serverAppConfigs/docs/Running Qwen3.6 27B FP8 on Dankclaw.md

Running Qwen3.6 27B FP8 on Dankclaw

Model Information

Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.

Property	Value
HuggingFace ID	Qwen/Qwen3.6-27B-FP8
Base Model	Qwen/Qwen3.6-27B
Architecture	Qwen3_5ForConditionalGeneration
Total Parameters	27B
Hidden Size	5120
Layers	64 (16 blocks of 3 DeltaNet + 1 Attention)
Attention Heads	24 Q, 4 KV (head dim 256)
DeltaNet Heads	48 V, 16 QK (head dim 128)
Intermediate Dim	17,408
Vocab Size	248,320 (padded)
Context Length	262,144 (natively extensible to 1,010,000)
Quantization	FP8 (block size 128)
Vision Encoder	Yes (image-text-to-text)

Hardware: Dankclaw

Property	Value
CPU	AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo)
GPU	Integrated Radeon 8060S (unified memory)
Physical RAM	30 GiB (shared CPU + GPU)
GPU-Visible (ROCm)	96 GiB (overcommitted)
Swap	8 GiB
ROCm	7.2.4
Python	3.12.13 (vLLM venv)

Memory Calculation

Component	BF16	FP8
Model weights (27B)	54 GB	13.5 GB
Vision encoder	~1 GB	~0.5 GB
Total weights	~55 GB	~14 GB
KV cache (64K ctx, 1 seq)	~1 GB	~0.5 GB
Overhead	~2 GB	~2 GB
Total (approximate)	~58 GB	~16.5 GB

The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB. Both fit within the 30 GiB physical RAM, but ROCm reports 96 GiB to vLLM. Set --gpu-memory-utilization conservatively (0.80) to avoid OOM.

Prerequisites

Production note: Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see scripts/frank-default-large.service). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.

Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Installed: vLLM 0.26.0 in Python 3.12.13 venv at /home/dank/vllm-rocm-venv.

MPI Dependency

Torch requires libmpi_cxx.so.40. If missing, install openmpi-bin:

sudo apt install -y openmpi-bin

Environment Variables

Set these for ROCm on Strix Halo:

export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface

vLLM Serve Commands

Text-Only (Recommended for Memory Savings)

Skip the vision encoder to reduce memory:

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --language-model-only \
    --dtype auto

With Vision Encoder

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto

Parameters Explained

Parameter	Value	Reason
--kv-cache-dtype fp8	FP8 KV cache	Cuts KV cache memory by 50%
--gpu-memory-utilization 0.85	85% of unified memory	Leaves 15% for CPU/OS
--swap-space 8	8 GB CPU swap	Matches system swap size
--max-model-len 131072	128K context	Full 262K is large but possible with 61 GB
--language-model-only	Skip vision	Saves ~1 GB for KV cache
--reasoning-parser qwen3	Qwen3 parser	Required for thinking mode

Sampling Parameters

Mode	Temperature	Top P	Top K	Min P	Presence Penalty
Thinking (general)	1.0	0.95	20	0.0	0.0
Thinking (coding)	0.6	0.95	20	0.0	0.0
Non-thinking	0.7	0.80	20	0.0	1.5

Set max_tokens to 32768 for general use. Set to 81920 for complex reasoning.

API Examples

Chat with Thinking

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'

Non-Thinking Mode

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'

Tool Calling

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288

ROCm Notes

• Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
• FP8 uses software emulation on RDNA architecture (gfx1151 has no native FP8 tensor cores). This gives memory savings, not full compute speedup. Compare against a BF16 test first
• Attention backends: try ROCM_AITER_UNIFIED_ATTN if the default backend underperforms
• Context length on ROCm may be limited for some models. Start with 131072 and increase gradually

Troubleshooting

Out of Memory

1. Use --language-model-only to skip vision encoder
2. Reduce --gpu-memory-utilization to 0.75
3. Reduce --max-model-len to 65536
4. Reduce --max-num-batched-tokens to 1024

FP8 Errors

1. Try BF16 first: vllm serve Qwen/Qwen3.6-27B --dtype bfloat16
2. Verify ROCm version supports FP8
3. Set HSA_OVERRIDE_GFX_VERSION=11.5.1
4. Use --kv-cache-dtype auto instead of fp8

Slow Inference

1. Check GPU utilization with rocm-smi
2. Verify attention backend is optimal
3. Increase --max-num-batched-tokens if memory allows
4. Ensure CPU cores are not throttled

See Also

• VLLM Overview
• Dankclaw System Specs
• Shared Memory Architecture Dankclaw
• RAM CPU Offloading in VLLM
• Running Qwen3.6 35B A3B FP8 on Dankserver