Update README with validation instructions and performance data

2026-08-08 21:46:00 -07:00
parent 726b9a35ec
commit 469bfaf979
2 changed files with 93 additions and 31 deletions
@@ -55,6 +55,34 @@ See [[docs/README.md]] for the project knowledge base index.
 - `--max-num-batched-tokens 1536`
 - Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`

+**Performance observations (no speculative decoding):**
+| Streams | Prompt Speed | Completion Speed | Total Time |
+|---------|-------------|-----------------|------------|
+| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
+| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
+| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
+| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
+
+Note: Speeds decrease with more streams due to KV cache contention.
+
 **Still needs work:**
 - DFlash speculative decoding integration (KV cache assertion bug in nightly)
 - Full context length support (32k is temporary workaround for OOM)
+
+## Validation
+
+Use `scripts/validate-vllm.sh` to test inference servers:
+
+```bash
+# Test dankserver (default)
+bash scripts/validate-vllm.sh -t dankserver
+
+# Test dankclaw
+bash scripts/validate-vllm.sh -t dankclaw
+
+# Custom host/port
+bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model
+
+# Run tests
+bash tests/test_validator.sh dankserver
+```
@@ -3,47 +3,81 @@
 ## Date: 2026-08-08

 ## Objective
-Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU (24GB VRAM).
+Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU.

-## Scripts Created
- `vllm-awq-qwen36-dflash.sh` — Docker-based AWQ + DFlash
- `vllm-awq-qwen36-dflash-ampere.sh` — Systemd service, Ampere-optimized
+## What Worked

-## Key Findings
+### Base AWQ Model
+- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully
+- AWQ Marlin kernel engaged automatically for weight conversion
+- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"` helps fit model in memory
+- `--gpu-memory-utilization 0.95` is safe for this model

-### vLLM CLI Changes (Nightly)
- `--speculative-model` removed; use `--speculative-config` JSON instead
- Format: `--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 4}'`
- `num_speculative_tokens` is required when model is provided
+### DFlash Draft Model
+- `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
+- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
+- Draft model shares embedding/lm_head weights with target model

-### Memory Issues
- **OOM with 262144 context**: AWQ→Marlin weight conversion requires extra GPU memory
- **Solution**: Lower to 32768 context temporarily
- **CPU offload**: `--cpu-offload-gb 40` + `--cpu-offload-params "experts"` helps
+### Performance (no speculative decoding)
+| Streams | Prompt Speed | Completion Speed | Total Time |
+|---------|-------------|-----------------|------------|
+| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
+| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
+| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
+| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |

-### Conflicts
- **Prefix caching + Mamba cache**: Causes `AssertionError` in `unify_kv_cache_spec_page_size`
- **DFlash speculative decoding**: KV cache assertion bug in nightly vLLM
- Workaround: Disable speculative decoding or prefix caching
+Note: Speeds decrease with more streams due to KV cache contention.

-### Performance Observations (no speculative decoding)
-| Streams | Prompt Speed | Completion Speed |
-|---------|-------------|------------------|
-| 1 | 4.27 tok/s | 21.42 tok/s |
-| 2 | 2.44 tok/s | 12.25 tok/s |
-| 3 | 1.35 tok/s | 6.76 tok/s |
-| 4 | 1.43 tok/s | 7.19 tok/s |
+## What Didn't Work

-Note: Speeds decrease with more streams due to KV cache and compute contention.
+### Prefix Caching + Mamba Cache Conflict
+- **Error**: `AssertionError` in `unify_kv_cache_spec_page_size`
+- **Cause**: Mamba cache "align" mode conflicts with prefix caching
+- **Fix**: Remove `--enable-prefix-caching` from command line
+
+### OOM with 262144 Context
+- **Error**: CUDA out of memory during AWQ→Marlin weight conversion
+- **Cause**: 24GB GPU cannot handle 262144 context + AWQ conversion
+- **Fix**: Use `--max-model-len 32768` (32k context)
+
+### Speculative Decoding KV Cache Bug
+- **Error**: `AssertionError` in nightly vLLM when using DFlash
+- **Cause**: Known bug in vLLM nightly with Mamba cache + speculative decoding
+- **Status**: Temporarily disabled speculative decoding to test base model

 ## Systemd Service
 - Service: `vllm-awq-qwen36.service`
 - ExecStart: `/home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh`
- Port: 34401
- Status: Disabled (needs speculative decoding fix)
+- Port: 34401 (to avoid conflicts with other services)
+- Status: Disabled (pending bug fixes)
+
+## Configuration Summary
+
+### Working Configuration
+```bash
+--max-model-len 32768
+--cpu-offload-gb 40
+--cpu-offload-params "experts"
+--gpu-memory-utilization 0.95
+--max-num-batched-tokens 1536
+--performance-mode throughput
+--enable-chunked-prefill
+# --enable-prefix-caching  # DISABLED - conflicts with Mamba
+```
+
+### Thread Binding
+```bash
+-e OMP_NUM_THREADS=8
+-e VLLM_CPU_OMP_THREADS_BIND=8
+```

 ## Next Steps
-1. Fix DFlash speculative decoding (upstream vLLM bug or workaround)
-2. Test with lower context length for full functionality
-3. Benchmark with and without speculative decoding
-4. Investigate `max_num_scheduled_tokens` warning (set to 1530 due to speculative settings)
+1. Monitor vLLM nightly for Mamba cache + speculative decoding fix
+2. Test with larger context windows when memory allows
+3. Benchmark with speculative decoding enabled (when bug is fixed)
+4. Consider using stable vLLM release instead of nightly
+
+## References
+- Model: https://huggingface.co/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
+- Draft: https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash
+- vLLM Docs: https://docs.vllm.ai/en/stable/