50 lines
1.9 KiB
Markdown
50 lines
1.9 KiB
Markdown
# AWQ + DFlash on Ampere Hardware
|
|
|
|
## Date: 2026-08-08
|
|
|
|
## Objective
|
|
Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU (24GB VRAM).
|
|
|
|
## Scripts Created
|
|
- `vllm-awq-qwen36-dflash.sh` — Docker-based AWQ + DFlash
|
|
- `vllm-awq-qwen36-dflash-ampere.sh` — Systemd service, Ampere-optimized
|
|
|
|
## Key Findings
|
|
|
|
### vLLM CLI Changes (Nightly)
|
|
- `--speculative-model` removed; use `--speculative-config` JSON instead
|
|
- Format: `--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 4}'`
|
|
- `num_speculative_tokens` is required when model is provided
|
|
|
|
### Memory Issues
|
|
- **OOM with 262144 context**: AWQ→Marlin weight conversion requires extra GPU memory
|
|
- **Solution**: Lower to 32768 context temporarily
|
|
- **CPU offload**: `--cpu-offload-gb 40` + `--cpu-offload-params "experts"` helps
|
|
|
|
### Conflicts
|
|
- **Prefix caching + Mamba cache**: Causes `AssertionError` in `unify_kv_cache_spec_page_size`
|
|
- **DFlash speculative decoding**: KV cache assertion bug in nightly vLLM
|
|
- Workaround: Disable speculative decoding or prefix caching
|
|
|
|
### Performance Observations (no speculative decoding)
|
|
| Streams | Prompt Speed | Completion Speed |
|
|
|---------|-------------|------------------|
|
|
| 1 | 4.27 tok/s | 21.42 tok/s |
|
|
| 2 | 2.44 tok/s | 12.25 tok/s |
|
|
| 3 | 1.35 tok/s | 6.76 tok/s |
|
|
| 4 | 1.43 tok/s | 7.19 tok/s |
|
|
|
|
Note: Speeds decrease with more streams due to KV cache and compute contention.
|
|
|
|
## Systemd Service
|
|
- Service: `vllm-awq-qwen36.service`
|
|
- ExecStart: `/home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh`
|
|
- Port: 34401
|
|
- Status: Disabled (needs speculative decoding fix)
|
|
|
|
## Next Steps
|
|
1. Fix DFlash speculative decoding (upstream vLLM bug or workaround)
|
|
2. Test with lower context length for full functionality
|
|
3. Benchmark with and without speculative decoding
|
|
4. Investigate `max_num_scheduled_tokens` warning (set to 1530 due to speculative settings)
|