84 lines
2.8 KiB
Markdown
84 lines
2.8 KiB
Markdown
# AWQ + DFlash on Ampere Hardware
|
|
|
|
## Date: 2026-08-08
|
|
|
|
## Objective
|
|
Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU.
|
|
|
|
## What Worked
|
|
|
|
### Base AWQ Model
|
|
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully
|
|
- AWQ Marlin kernel engaged automatically for weight conversion
|
|
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"` helps fit model in memory
|
|
- `--gpu-memory-utilization 0.95` is safe for this model
|
|
|
|
### DFlash Draft Model
|
|
- `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
|
|
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
|
|
- Draft model shares embedding/lm_head weights with target model
|
|
|
|
### Performance (no speculative decoding)
|
|
| Streams | Prompt Speed | Completion Speed | Total Time |
|
|
|---------|-------------|-----------------|------------|
|
|
| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
|
|
| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
|
|
| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
|
|
| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
|
|
|
|
Note: Speeds decrease with more streams due to KV cache contention.
|
|
|
|
## What Didn't Work
|
|
|
|
### Prefix Caching + Mamba Cache Conflict
|
|
- **Error**: `AssertionError` in `unify_kv_cache_spec_page_size`
|
|
- **Cause**: Mamba cache "align" mode conflicts with prefix caching
|
|
- **Fix**: Remove `--enable-prefix-caching` from command line
|
|
|
|
### OOM with 262144 Context
|
|
- **Error**: CUDA out of memory during AWQ→Marlin weight conversion
|
|
- **Cause**: 24GB GPU cannot handle 262144 context + AWQ conversion
|
|
- **Fix**: Use `--max-model-len 32768` (32k context)
|
|
|
|
### Speculative Decoding KV Cache Bug
|
|
- **Error**: `AssertionError` in nightly vLLM when using DFlash
|
|
- **Cause**: Known bug in vLLM nightly with Mamba cache + speculative decoding
|
|
- **Status**: Temporarily disabled speculative decoding to test base model
|
|
|
|
## Systemd Service
|
|
- Service: `vllm-awq-qwen36.service`
|
|
- ExecStart: `/home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh`
|
|
- Port: 34401 (to avoid conflicts with other services)
|
|
- Status: Disabled (pending bug fixes)
|
|
|
|
## Configuration Summary
|
|
|
|
### Working Configuration
|
|
```bash
|
|
--max-model-len 32768
|
|
--cpu-offload-gb 40
|
|
--cpu-offload-params "experts"
|
|
--gpu-memory-utilization 0.95
|
|
--max-num-batched-tokens 1536
|
|
--performance-mode throughput
|
|
--enable-chunked-prefill
|
|
# --enable-prefix-caching # DISABLED - conflicts with Mamba
|
|
```
|
|
|
|
### Thread Binding
|
|
```bash
|
|
-e OMP_NUM_THREADS=8
|
|
-e VLLM_CPU_OMP_THREADS_BIND=8
|
|
```
|
|
|
|
## Next Steps
|
|
1. Monitor vLLM nightly for Mamba cache + speculative decoding fix
|
|
2. Test with larger context windows when memory allows
|
|
3. Benchmark with speculative decoding enabled (when bug is fixed)
|
|
4. Consider using stable vLLM release instead of nightly
|
|
|
|
## References
|
|
- Model: https://huggingface.co/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
|
|
- Draft: https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash
|
|
- vLLM Docs: https://docs.vllm.ai/en/stable/
|