serverAppConfigs/docs/Running Qwen3.6 27B FP8 on Dankclaw.md

# Running Qwen3.6 27B FP8 on Dankclaw

## Model Information

Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.

| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-27B-FP8` |
| Base Model | `Qwen/Qwen3.6-27B` |
| Architecture | Qwen3_5ForConditionalGeneration |
| Total Parameters | 27B |
| Hidden Size | 5120 |
| Layers | 64 (16 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 24 Q, 4 KV (head dim 256) |
| DeltaNet Heads | 48 V, 16 QK (head dim 128) |
| Intermediate Dim | 17,408 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |

## Hardware: Dankclaw

| Property | Value |
|----------|-------|
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
| GPU | Integrated Radeon 8060S (unified memory) |
| Physical RAM | 64 GiB (61 GiB visible; corrected 2026-08-17) |
| GPU-Visible (ROCm) | 96 GiB (overcommitted) |
| Swap | 8 GiB |
| ROCm | 7.2.4 |
| Python | 3.12.13 (vLLM venv) |

## Memory Calculation

| Component | BF16 | FP8 |
|-----------|------|-----|
| Model weights (27B) | 54 GB | 13.5 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~55 GB | ~14 GB |
| KV cache (64K ctx, 1 seq) | ~1 GB | ~0.5 GB |
| Overhead | ~2 GB | ~2 GB |
| **Total (approximate)** | ~58 GB | ~16.5 GB |

The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB. Both fit within the 61 GiB physical RAM, but ROCm reports 96 GiB to vLLM. Set `--gpu-memory-utilization` conservatively (0.80) to avoid OOM.

## Prerequisites

> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.

Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:

```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

Installed: vLLM 0.26.0 in Python 3.12.13 venv at `/home/dank/vllm-rocm-venv`.

### MPI Dependency

Torch requires `libmpi_cxx.so.40`. If missing, install `openmpi-bin`:

```
sudo apt install -y openmpi-bin
```

### Environment Variables

Set these for ROCm on Strix Halo:

```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface
```

## vLLM Serve Commands

### Text-Only (Recommended for Memory Savings)

Skip the vision encoder to reduce memory:

```
vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --language-model-only \
    --dtype auto
```

### With Vision Encoder

```
vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto
```

### Parameters Explained

| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.85 | 85% of unified memory | Leaves 15% for CPU/OS |
| --swap-space 8 | 8 GB CPU swap | Matches system swap size |
| --max-model-len 131072 | 128K context | Full 262K is large but possible with 61 GB |
| --language-model-only | Skip vision | Saves ~1 GB for KV cache |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode |

## Sampling Parameters

| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |

Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.

## API Examples

### Chat with Thinking

```
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'
```

### Non-Thinking Mode

```
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-27B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'
```

### Tool Calling

```
vllm serve Qwen/Qwen3.6-27B-FP8 \
    --port 8000 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder
```

## Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens:

```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288
```

## ROCm Notes

- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
- FP8 uses **software emulation** on RDNA architecture (gfx1151 has no native FP8 tensor cores). This gives memory savings, not full compute speedup. Compare against a BF16 test first
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually

## Troubleshooting

### Out of Memory

1. Use `--language-model-only` to skip vision encoder
2. Reduce `--gpu-memory-utilization` to 0.75
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 1024

### FP8 Errors

1. Try BF16 first: `vllm serve Qwen/Qwen3.6-27B --dtype bfloat16`
2. Verify ROCm version supports FP8
3. Set `HSA_OVERRIDE_GFX_VERSION=11.5.1`
4. Use `--kv-cache-dtype auto` instead of fp8

### Slow Inference

1. Check GPU utilization with `rocm-smi`
2. Verify attention backend is optimal
3. Increase `--max-num-batched-tokens` if memory allows
4. Ensure CPU cores are not throttled

## See Also

- [[VLLM Overview]]
- [[Dankclaw System Specs]]
- [[Shared Memory Architecture Dankclaw]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]