serverAppConfigs/docs/awq-dflash-ampere-notes.md

AWQ + DFlash on Ampere Hardware

Date: 2026-08-08

Objective

Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU.

What Worked

Base AWQ Model

• cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit loads successfully
• AWQ Marlin kernel engaged automatically for weight conversion
• --cpu-offload-gb 40 with --cpu-offload-params "experts" helps fit model in memory
• --gpu-memory-utilization 0.95 is safe for this model

DFlash Draft Model

• z-lab/Qwen3.6-35B-A3B-DFlash detected as EAGLE model
• torch.compile with cudagraphs works with --max-num-batched-tokens 1536
• Draft model shares embedding/lm_head weights with target model

Performance (no speculative decoding)

Streams	Prompt Speed	Completion Speed	Total Time
1	4.27 tok/s	21.42 tok/s	11,951ms
2	2.44 tok/s	12.25 tok/s	20,891ms
3	1.35 tok/s	6.76 tok/s	37,849ms
4	1.43 tok/s	7.19 tok/s	35,606ms

Note: Speeds decrease with more streams due to KV cache contention.

What Didn't Work

Prefix Caching + Mamba Cache Conflict

• Error: AssertionError in unify_kv_cache_spec_page_size
• Cause: Mamba cache "align" mode conflicts with prefix caching
• Fix: Remove --enable-prefix-caching from command line

OOM with 262144 Context

• Error: CUDA out of memory during AWQ→Marlin weight conversion
• Cause: 24GB GPU cannot handle 262144 context + AWQ conversion
• Fix: Use --max-model-len 32768 (32k context)

Speculative Decoding KV Cache Bug

• Error: AssertionError in nightly vLLM when using DFlash
• Cause: Known bug in vLLM nightly with Mamba cache + speculative decoding
• Status: Temporarily disabled speculative decoding to test base model

Systemd Service

• Service: vllm-awq-qwen36.service
• ExecStart: /home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh
• Port: 34401 (to avoid conflicts with other services)
• Status: Disabled (pending bug fixes)

Configuration Summary

Working Configuration

--max-model-len 32768
--cpu-offload-gb 40
--cpu-offload-params "experts"
--gpu-memory-utilization 0.95
--max-num-batched-tokens 1536
--performance-mode throughput
--enable-chunked-prefill
# --enable-prefix-caching  # DISABLED - conflicts with Mamba

Thread Binding

-e OMP_NUM_THREADS=8
-e VLLM_CPU_OMP_THREADS_BIND=8

Next Steps

1. Monitor vLLM nightly for Mamba cache + speculative decoding fix
2. Test with larger context windows when memory allows
3. Benchmark with speculative decoding enabled (when bug is fixed)
4. Consider using stable vLLM release instead of nightly

References

• Model: https://huggingface.co/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
• Draft: https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash
• vLLM Docs: https://docs.vllm.ai/en/stable/