serverAppConfigs/docs/awq-dflash-ampere-notes.md

AWQ + DFlash on Ampere Hardware

Date: 2026-08-08

Objective

Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU (24GB VRAM).

Scripts Created

• vllm-awq-qwen36-dflash.sh — Docker-based AWQ + DFlash
• vllm-awq-qwen36-dflash-ampere.sh — Systemd service, Ampere-optimized

Key Findings

vLLM CLI Changes (Nightly)

• --speculative-model removed; use --speculative-config JSON instead
• Format: --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 4}'
• num_speculative_tokens is required when model is provided

Memory Issues

• OOM with 262144 context: AWQ→Marlin weight conversion requires extra GPU memory
• Solution: Lower to 32768 context temporarily
• CPU offload: --cpu-offload-gb 40 + --cpu-offload-params "experts" helps

Conflicts

• Prefix caching + Mamba cache: Causes AssertionError in unify_kv_cache_spec_page_size
• DFlash speculative decoding: KV cache assertion bug in nightly vLLM
• Workaround: Disable speculative decoding or prefix caching

Performance Observations (no speculative decoding)

Streams	Prompt Speed	Completion Speed
1	4.27 tok/s	21.42 tok/s
2	2.44 tok/s	12.25 tok/s
3	1.35 tok/s	6.76 tok/s
4	1.43 tok/s	7.19 tok/s

Note: Speeds decrease with more streams due to KV cache and compute contention.

Systemd Service

• Service: vllm-awq-qwen36.service
• ExecStart: /home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh
• Port: 34401
• Status: Disabled (needs speculative decoding fix)

Next Steps

1. Fix DFlash speculative decoding (upstream vLLM bug or workaround)
2. Test with lower context length for full functionality
3. Benchmark with and without speculative decoding
4. Investigate max_num_scheduled_tokens warning (set to 1530 due to speculative settings)