serverAppConfigs/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md

# Running Qwen3.6 35B A3B FP8 on Dankserver

> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].

## Model Information

Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.

| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-35B-A3B-FP8` |
| Base Model | `Qwen/Qwen3.6-35B-A3B` |
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Total Parameters | 35B |
| Activated Parameters | 3B |
| Experts | 256 (8 routed + 1 shared per token) |
| Layers | 40 (10 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 16 Q, 2 KV (head dim 256) |
| DeltaNet Heads | 32 V, 16 QK (head dim 128) |
| Expert Intermediate Dim | 512 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |

## Hardware: Dankserver

| Property | Value |
|----------|-------|
| GPU | NVIDIA GeForce RTX 3090 |
| VRAM | 24 GB |
| System RAM | 62 GB |
| Swap | 127 GB |

## Memory Calculation

| Component | BF16 | FP8 |
|-----------|------|-----|
| Activated weights (3B) | 6 GB | 1.5 GB |
| Inactive experts (32B) | 64 GB | 16 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~71 GB | ~18 GB |

The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.

## Prerequisites

Install vLLM. Version 0.19.0 or later is required for Qwen3.6:

```
uv pip install vllm --torch-backend=auto
```

## Hardware Quantization Notes (RTX 3090 / Ampere)

The RTX 3090 is an Ampere GPU (compute capability 8.6). It has **no native FP8 tensor cores**. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. `--kv-cache-dtype fp8` is still valid on CUDA 12 for an FP8 KV cache.

> Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.

## vLLM Serve Commands

### Text-Only (Recommended for Memory Savings)

Skip the vision encoder to free VRAM for KV cache:

```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.75 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32 \
    --reasoning-parser qwen3 \
    --dtype auto
```

### With Vision Encoder

If you need multimodal input, expect higher VRAM usage:

```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --port 8000 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 131072 \
    --max-num-batched-tokens 1024 \
    --max-num-seqs 16 \
    --reasoning-parser qwen3 \
    --dtype auto
```

### Parameters Explained

| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
| --swap-space 32 (removed) | 32 GB CPU overflow | Modern vLLM removed swap-space; use --kv-offloading-size |
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |

### Model Storage

Root disk has only 20 GB free. Store models on /mnt/entropy:

```
export HF_HOME=/mnt/entropy/huggingface
```

### Active Service Conflict

The retired `frank-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:

```
nvidia-smi  # verify GPU is free
```

## Sampling Parameters

| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |

Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.

## API Examples

### Chat with Thinking

```
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Solve: 2+2"}],
        "max_tokens": 32768,
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20
    }'
```

### Non-Thinking Mode

```
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
        "messages": [{"role": "user", "content": "Hello"}],
        "max_tokens": 2048,
        "temperature": 0.7,
        "top_p": 0.8,
        "top_k": 20,
        "presence_penalty": 1.5,
        "extra_body": {
            "chat_template_kwargs": {"enable_thinking": false}
        }
    }'
```

## Ultra-Long Context (YaRN)

Extend context beyond 262,144 tokens. Use this environment variable and override:

```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
    --max-model-len 524288
```

## Troubleshooting

### Out of Memory

1. Kill llama-server to free 20 GB VRAM
2. Use `--language-model-only` to skip vision encoder
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 512
5. Reduce `--max-num-seqs` to 8

### Slow Inference

1. Verify GPU is exclusive to vLLM
2. Check temperature with `nvidia-smi`
3. Ensure `--kv-cache-dtype fp8` is set
4. Increase `--max-num-batched-tokens` if memory allows

## See Also

- [[VLLM Overview]]
- [[Dankserver System Specs]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]