serverAppConfigs/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md

Running Qwen3.6 35B A3B FP8 on Dankserver

Model Information

Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.

Property	Value
HuggingFace ID	Qwen/Qwen3.6-35B-A3B-FP8
Base Model	Qwen/Qwen3.6-35B-A3B
Architecture	Qwen3_5MoeForConditionalGeneration
Total Parameters	35B
Activated Parameters	3B
Experts	256 (8 routed + 1 shared per token)
Layers	40 (10 blocks of 3 DeltaNet + 1 Attention)
Attention Heads	16 Q, 2 KV (head dim 256)
DeltaNet Heads	32 V, 16 QK (head dim 128)
Expert Intermediate Dim	512
Vocab Size	248,320 (padded)
Context Length	262,144 (natively extensible to 1,010,000)
Quantization	FP8 (block size 128)
Vision Encoder	Yes (image-text-to-text)

Hardware: Dankserver

Property	Value
GPU	NVIDIA GeForce RTX 3090
VRAM	24 GB
System RAM	62 GB
Swap	127 GB

Memory Calculation

Component	BF16	FP8
Activated weights (3B)	6 GB	1.5 GB
Inactive experts (32B)	64 GB	16 GB
Vision encoder	~1 GB	~0.5 GB
Total weights	~71 GB	~18 GB

The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.

Prerequisites

Install vLLM. Version 0.19.0 or later is required for Qwen3.6:

uv pip install vllm --torch-backend=auto

vLLM Serve Commands

Text-Only (Recommended for Memory Savings)

Skip the vision encoder to free VRAM for KV cache:

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.75 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto

With Vision Encoder

If you need multimodal input, expect higher VRAM usage:

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 1024 \
    --max-num-seqs 16 \
    --reasoning-parser qwen3 \
    --dtype auto

Parameters Explained

Parameter	Value	Reason
--kv-cache-dtype fp8	FP8 KV cache	Cuts KV cache memory by 50%
--gpu-memory-utilization 0.75	75% of VRAM	Leaves room for all MoE weights
--swap-space 32	32 GB CPU swap	Dankserver has 62 GB RAM
--max-model-len 131072	128K context	Full 262K context is too large for 24 GB
--max-num-batched-tokens 2048	Conservative batch	Prevents OOM with MoE
--reasoning-parser qwen3	Qwen3 parser	Required for thinking mode output

Model Storage

Root disk has only 20 GB free. Store models on /mnt/entropy:

export HF_HOME=/mnt/entropy/huggingface

Active Service Conflict

A llama-server process occupies approximately 20 GB of VRAM. Stop it before starting vLLM:

pkill llama-server
nvidia-smi  # verify GPU is free

Sampling Parameters

Mode	Temperature	Top P	Top K	Min P	Presence Penalty
Thinking (general)	1.0	0.95	20	0.0	0.0
Thinking (coding)	0.6	0.95	20	0.0	0.0
Non-thinking	0.7	0.80	20	0.0	1.5

Set max_tokens to 32768 for general use. Set to 81920 for complex reasoning.

API Examples

Chat with Thinking

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'

Non-Thinking Mode

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'

Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens. Use this environment variable and override:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288

Troubleshooting

Out of Memory

1. Kill llama-server to free 20 GB VRAM
2. Use --language-model-only to skip vision encoder
3. Reduce --max-model-len to 65536
4. Reduce --max-num-batched-tokens to 512
5. Reduce --max-num-seqs to 8

Slow Inference

1. Verify GPU is exclusive to vLLM
2. Check temperature with nvidia-smi
3. Ensure --kv-cache-dtype fp8 is set
4. Increase --max-num-batched-tokens if memory allows

See Also

• VLLM Overview
• Dankserver System Specs
• Hybrid CPU GPU Processing Dankserver
• RAM CPU Offloading in VLLM
• Running Qwen3.6 27B FP8 on Dankclaw