serverAppConfigs/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md

Running Qwen3.6 35B A3B FP8 on Dankserver

RETIRED. This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (vllm-qwen36) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see Qwen3.8 PP Deployment.

Model Information

Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.

Property	Value
HuggingFace ID	Qwen/Qwen3.6-35B-A3B-FP8
Base Model	Qwen/Qwen3.6-35B-A3B
Architecture	Qwen3_5MoeForConditionalGeneration
Total Parameters	35B
Activated Parameters	3B
Experts	256 (8 routed + 1 shared per token)
Layers	40 (10 blocks of 3 DeltaNet + 1 Attention)
Attention Heads	16 Q, 2 KV (head dim 256)
DeltaNet Heads	32 V, 16 QK (head dim 128)
Expert Intermediate Dim	512
Vocab Size	248,320 (padded)
Context Length	262,144 (natively extensible to 1,010,000)
Quantization	FP8 (block size 128)
Vision Encoder	Yes (image-text-to-text)

Hardware: Dankserver

Property	Value
GPU	NVIDIA GeForce RTX 3090
VRAM	24 GB
System RAM	62 GB
Swap	127 GB

Memory Calculation

Component	BF16	FP8
Activated weights (3B)	6 GB	1.5 GB
Inactive experts (32B)	64 GB	16 GB
Vision encoder	~1 GB	~0.5 GB
Total weights	~71 GB	~18 GB

The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.

Prerequisites

Install vLLM. Version 0.19.0 or later is required for Qwen3.6:

uv pip install vllm --torch-backend=auto

Hardware Quantization Notes (RTX 3090 / Ampere)

The RTX 3090 is an Ampere GPU (compute capability 8.6). It has no native FP8 tensor cores. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. --kv-cache-dtype fp8 is still valid on CUDA 12 for an FP8 KV cache.

Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.

vLLM Serve Commands

Text-Only (Recommended for Memory Savings)

Skip the vision encoder to free VRAM for KV cache:

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.75 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto

With Vision Encoder

If you need multimodal input, expect higher VRAM usage:

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 1024 \
    --max-num-seqs 16 \
    --reasoning-parser qwen3 \
    --dtype auto

Parameters Explained

Parameter	Value	Reason
--kv-cache-dtype fp8	FP8 KV cache	Cuts KV cache memory by 50%
--gpu-memory-utilization 0.75	75% of VRAM	Leaves room for all MoE weights
--swap-space 32 (removed)	32 GB CPU overflow	Modern vLLM removed swap-space; use --kv-offloading-size
--max-model-len 131072	128K context	Full 262K context is too large for 24 GB
--max-num-batched-tokens 2048	Conservative batch	Prevents OOM with MoE
--reasoning-parser qwen3	Qwen3 parser	Required for thinking mode output

Model Storage

Root disk has only 20 GB free. Store models on /mnt/entropy:

export HF_HOME=/mnt/entropy/huggingface

Active Service Conflict

The retired frank-small service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The vllm-awq-qwen36 (Podman) and vllm-qwen36 units are disabled. Before starting any research service, verify that the GPU is free:

nvidia-smi  # verify GPU is free

Sampling Parameters

Mode	Temperature	Top P	Top K	Min P	Presence Penalty
Thinking (general)	1.0	0.95	20	0.0	0.0
Thinking (coding)	0.6	0.95	20	0.0	0.0
Non-thinking	0.7	0.80	20	0.0	1.5

Set max_tokens to 32768 for general use. Set to 81920 for complex reasoning.

API Examples

Chat with Thinking

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'

Non-Thinking Mode

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'

Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens. Use this environment variable and override:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288

Troubleshooting

Out of Memory

1. Kill llama-server to free 20 GB VRAM
2. Use --language-model-only to skip vision encoder
3. Reduce --max-model-len to 65536
4. Reduce --max-num-batched-tokens to 512
5. Reduce --max-num-seqs to 8

Slow Inference

1. Verify GPU is exclusive to vLLM
2. Check temperature with nvidia-smi
3. Ensure --kv-cache-dtype fp8 is set
4. Increase --max-num-batched-tokens if memory allows

See Also

• VLLM Overview
• Dankserver System Specs
• Hybrid CPU GPU Processing Dankserver
• RAM CPU Offloading in VLLM
• Running Qwen3.6 27B FP8 on Dankclaw