diff --git a/README.md b/README.md index df0bd64..4dbcdbf 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,34 @@ See [[docs/README.md]] for the project knowledge base index. - `--max-num-batched-tokens 1536` - Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8` +**Performance observations (no speculative decoding):** +| Streams | Prompt Speed | Completion Speed | Total Time | +|---------|-------------|-----------------|------------| +| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms | +| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms | +| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms | +| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms | + +Note: Speeds decrease with more streams due to KV cache contention. + **Still needs work:** - DFlash speculative decoding integration (KV cache assertion bug in nightly) - Full context length support (32k is temporary workaround for OOM) + +## Validation + +Use `scripts/validate-vllm.sh` to test inference servers: + +```bash +# Test dankserver (default) +bash scripts/validate-vllm.sh -t dankserver + +# Test dankclaw +bash scripts/validate-vllm.sh -t dankclaw + +# Custom host/port +bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model + +# Run tests +bash tests/test_validator.sh dankserver +``` diff --git a/docs/awq-dflash-ampere-notes.md b/docs/awq-dflash-ampere-notes.md index d32918b..dc0a1d2 100644 --- a/docs/awq-dflash-ampere-notes.md +++ b/docs/awq-dflash-ampere-notes.md @@ -3,47 +3,81 @@ ## Date: 2026-08-08 ## Objective -Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU (24GB VRAM). +Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU. -## Scripts Created -- `vllm-awq-qwen36-dflash.sh` — Docker-based AWQ + DFlash -- `vllm-awq-qwen36-dflash-ampere.sh` — Systemd service, Ampere-optimized +## What Worked -## Key Findings +### Base AWQ Model +- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully +- AWQ Marlin kernel engaged automatically for weight conversion +- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"` helps fit model in memory +- `--gpu-memory-utilization 0.95` is safe for this model -### vLLM CLI Changes (Nightly) -- `--speculative-model` removed; use `--speculative-config` JSON instead -- Format: `--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 4}'` -- `num_speculative_tokens` is required when model is provided +### DFlash Draft Model +- `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model +- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536` +- Draft model shares embedding/lm_head weights with target model -### Memory Issues -- **OOM with 262144 context**: AWQ→Marlin weight conversion requires extra GPU memory -- **Solution**: Lower to 32768 context temporarily -- **CPU offload**: `--cpu-offload-gb 40` + `--cpu-offload-params "experts"` helps +### Performance (no speculative decoding) +| Streams | Prompt Speed | Completion Speed | Total Time | +|---------|-------------|-----------------|------------| +| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms | +| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms | +| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms | +| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms | -### Conflicts -- **Prefix caching + Mamba cache**: Causes `AssertionError` in `unify_kv_cache_spec_page_size` -- **DFlash speculative decoding**: KV cache assertion bug in nightly vLLM -- Workaround: Disable speculative decoding or prefix caching +Note: Speeds decrease with more streams due to KV cache contention. -### Performance Observations (no speculative decoding) -| Streams | Prompt Speed | Completion Speed | -|---------|-------------|------------------| -| 1 | 4.27 tok/s | 21.42 tok/s | -| 2 | 2.44 tok/s | 12.25 tok/s | -| 3 | 1.35 tok/s | 6.76 tok/s | -| 4 | 1.43 tok/s | 7.19 tok/s | +## What Didn't Work -Note: Speeds decrease with more streams due to KV cache and compute contention. +### Prefix Caching + Mamba Cache Conflict +- **Error**: `AssertionError` in `unify_kv_cache_spec_page_size` +- **Cause**: Mamba cache "align" mode conflicts with prefix caching +- **Fix**: Remove `--enable-prefix-caching` from command line + +### OOM with 262144 Context +- **Error**: CUDA out of memory during AWQ→Marlin weight conversion +- **Cause**: 24GB GPU cannot handle 262144 context + AWQ conversion +- **Fix**: Use `--max-model-len 32768` (32k context) + +### Speculative Decoding KV Cache Bug +- **Error**: `AssertionError` in nightly vLLM when using DFlash +- **Cause**: Known bug in vLLM nightly with Mamba cache + speculative decoding +- **Status**: Temporarily disabled speculative decoding to test base model ## Systemd Service - Service: `vllm-awq-qwen36.service` - ExecStart: `/home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh` -- Port: 34401 -- Status: Disabled (needs speculative decoding fix) +- Port: 34401 (to avoid conflicts with other services) +- Status: Disabled (pending bug fixes) + +## Configuration Summary + +### Working Configuration +```bash +--max-model-len 32768 +--cpu-offload-gb 40 +--cpu-offload-params "experts" +--gpu-memory-utilization 0.95 +--max-num-batched-tokens 1536 +--performance-mode throughput +--enable-chunked-prefill +# --enable-prefix-caching # DISABLED - conflicts with Mamba +``` + +### Thread Binding +```bash +-e OMP_NUM_THREADS=8 +-e VLLM_CPU_OMP_THREADS_BIND=8 +``` ## Next Steps -1. Fix DFlash speculative decoding (upstream vLLM bug or workaround) -2. Test with lower context length for full functionality -3. Benchmark with and without speculative decoding -4. Investigate `max_num_scheduled_tokens` warning (set to 1530 due to speculative settings) +1. Monitor vLLM nightly for Mamba cache + speculative decoding fix +2. Test with larger context windows when memory allows +3. Benchmark with speculative decoding enabled (when bug is fixed) +4. Consider using stable vLLM release instead of nightly + +## References +- Model: https://huggingface.co/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit +- Draft: https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash +- vLLM Docs: https://docs.vllm.ai/en/stable/