Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
211 lines
6.6 KiB
Markdown
211 lines
6.6 KiB
Markdown
---
|
|
type: reference
|
|
description: "Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series."
|
|
tags:
|
|
- llm
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
# Running Qwen3.6 35B A3B FP8 on Dankserver
|
|
|
|
> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
|
|
|
|
## Model Information
|
|
|
|
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| HuggingFace ID | `Qwen/Qwen3.6-35B-A3B-FP8` |
|
|
| Base Model | `Qwen/Qwen3.6-35B-A3B` |
|
|
| Architecture | Qwen3_5MoeForConditionalGeneration |
|
|
| Total Parameters | 35B |
|
|
| Activated Parameters | 3B |
|
|
| Experts | 256 (8 routed + 1 shared per token) |
|
|
| Layers | 40 (10 blocks of 3 DeltaNet + 1 Attention) |
|
|
| Attention Heads | 16 Q, 2 KV (head dim 256) |
|
|
| DeltaNet Heads | 32 V, 16 QK (head dim 128) |
|
|
| Expert Intermediate Dim | 512 |
|
|
| Vocab Size | 248,320 (padded) |
|
|
| Context Length | 262,144 (natively extensible to 1,010,000) |
|
|
| Quantization | FP8 (block size 128) |
|
|
| Vision Encoder | Yes (image-text-to-text) |
|
|
|
|
## Hardware: Dankserver
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| GPU | NVIDIA GeForce RTX 3090 |
|
|
| VRAM | 24 GB |
|
|
| System RAM | 62 GB |
|
|
| Swap | 127 GB |
|
|
|
|
## Memory Calculation
|
|
|
|
| Component | BF16 | FP8 |
|
|
|-----------|------|-----|
|
|
| Activated weights (3B) | 6 GB | 1.5 GB |
|
|
| Inactive experts (32B) | 64 GB | 16 GB |
|
|
| Vision encoder | ~1 GB | ~0.5 GB |
|
|
| **Total weights** | ~71 GB | ~18 GB |
|
|
|
|
The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.
|
|
|
|
## Prerequisites
|
|
|
|
Install vLLM. Version 0.19.0 or later is required for Qwen3.6:
|
|
|
|
```
|
|
uv pip install vllm --torch-backend=auto
|
|
```
|
|
|
|
## Hardware Quantization Notes (RTX 3090 / Ampere)
|
|
|
|
The RTX 3090 is an Ampere GPU (compute capability 8.6). It has **no native FP8 tensor cores**. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. `--kv-cache-dtype fp8` is still valid on CUDA 12 for an FP8 KV cache.
|
|
|
|
> Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.
|
|
|
|
## vLLM Serve Commands
|
|
|
|
### Text-Only (Recommended for Memory Savings)
|
|
|
|
Skip the vision encoder to free VRAM for KV cache:
|
|
|
|
```
|
|
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
|
--port 8000 \
|
|
--kv-cache-dtype fp8 \
|
|
--gpu-memory-utilization 0.75 \
|
|
--swap-space 32 \
|
|
--max-model-len 131072 \
|
|
--max-num-batched-tokens 2048 \
|
|
--max-num-seqs 32 \
|
|
--reasoning-parser qwen3 \
|
|
--dtype auto
|
|
```
|
|
|
|
### With Vision Encoder
|
|
|
|
If you need multimodal input, expect higher VRAM usage:
|
|
|
|
```
|
|
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
|
--port 8000 \
|
|
--kv-cache-dtype fp8 \
|
|
--gpu-memory-utilization 0.7 \
|
|
--swap-space 32 \
|
|
--max-model-len 131072 \
|
|
--max-num-batched-tokens 1024 \
|
|
--max-num-seqs 16 \
|
|
--reasoning-parser qwen3 \
|
|
--dtype auto
|
|
```
|
|
|
|
### Parameters Explained
|
|
|
|
| Parameter | Value | Reason |
|
|
|-----------|-------|--------|
|
|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
|
|
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
|
|
| --swap-space 32 (removed) | 32 GB CPU overflow | Modern vLLM removed swap-space; use --kv-offloading-size |
|
|
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
|
|
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
|
|
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |
|
|
|
|
### Model Storage
|
|
|
|
Root disk has only 20 GB free. Store models on /mnt/entropy:
|
|
|
|
```
|
|
export HF_HOME=/mnt/entropy/huggingface
|
|
```
|
|
|
|
### Active Service Conflict
|
|
|
|
The retired `frank-zealot` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
|
|
|
```
|
|
nvidia-smi # verify GPU is free
|
|
```
|
|
|
|
## Sampling Parameters
|
|
|
|
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|
|
|------|-------------|-------|-------|-------|-----------------|
|
|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
|
|
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
|
|
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
|
|
|
|
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
|
|
|
|
## API Examples
|
|
|
|
### Chat with Thinking
|
|
|
|
```
|
|
curl http://localhost:8000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
|
|
"messages": [{"role": "user", "content": "Solve: 2+2"}],
|
|
"max_tokens": 32768,
|
|
"temperature": 1.0,
|
|
"top_p": 0.95,
|
|
"top_k": 20
|
|
}'
|
|
```
|
|
|
|
### Non-Thinking Mode
|
|
|
|
```
|
|
curl http://localhost:8000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
|
|
"messages": [{"role": "user", "content": "Hello"}],
|
|
"max_tokens": 2048,
|
|
"temperature": 0.7,
|
|
"top_p": 0.8,
|
|
"top_k": 20,
|
|
"presence_penalty": 1.5,
|
|
"extra_body": {
|
|
"chat_template_kwargs": {"enable_thinking": false}
|
|
}
|
|
}'
|
|
```
|
|
|
|
## Ultra-Long Context (YaRN)
|
|
|
|
Extend context beyond 262,144 tokens. Use this environment variable and override:
|
|
|
|
```
|
|
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
|
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
|
|
--max-model-len 524288
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Out of Memory
|
|
|
|
1. Kill llama-server to free 20 GB VRAM
|
|
2. Use `--language-model-only` to skip vision encoder
|
|
3. Reduce `--max-model-len` to 65536
|
|
4. Reduce `--max-num-batched-tokens` to 512
|
|
5. Reduce `--max-num-seqs` to 8
|
|
|
|
### Slow Inference
|
|
|
|
1. Verify GPU is exclusive to vLLM
|
|
2. Check temperature with `nvidia-smi`
|
|
3. Ensure `--kv-cache-dtype fp8` is set
|
|
4. Increase `--max-num-batched-tokens` if memory allows
|
|
|
|
## See Also
|
|
|
|
- [[VLLM Overview]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Hybrid CPU GPU Processing Dankserver]]
|
|
- [[RAM CPU Offloading in VLLM]]
|
|
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|