89 lines
3.1 KiB
Markdown
89 lines
3.1 KiB
Markdown
# vLLM Scripts
|
|
|
|
Scripts for running large language models via vLLM on two local machines.
|
|
|
|
## Targets
|
|
|
|
| Host | User | Codename | Architecture |
|
|
|------|------|----------|-------------|
|
|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
|
|
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
|
|
|
|
## Models
|
|
|
|
| Model | Target | Format |
|
|
|-------|--------|--------|
|
|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
|
|
| Qwen3.6 27B FP8 | dankclaw | FP8 |
|
|
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
|
|
| Qwen3.6 35B A3B DFlash | dankserver | Draft model |
|
|
|
|
## Structure
|
|
|
|
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
|
|
- `scripts/` — Executable vLLM launch scripts
|
|
- `vllm-qwen36.sh` — FP8 inference script for dankserver
|
|
- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
|
|
- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)
|
|
|
|
## Knowledge Base
|
|
|
|
See [[docs/README.md]] for the project knowledge base index.
|
|
|
|
## Lessons Learned
|
|
|
|
### AWQ + DFlash on Ampere Hardware (2026-08-08)
|
|
|
|
**What worked:**
|
|
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
|
|
- AWQ Marlin kernel engaged automatically for weight conversion
|
|
- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
|
|
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
|
|
|
|
**What didn't work:**
|
|
- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
|
|
- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
|
|
- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
|
|
- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead
|
|
|
|
**Working configuration:**
|
|
- `--max-model-len 32768` (32k context)
|
|
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
|
|
- `--gpu-memory-utilization 0.95`
|
|
- `--performance-mode throughput`
|
|
- `--enable-chunked-prefill`
|
|
- `--max-num-batched-tokens 1536`
|
|
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
|
|
|
|
**Performance observations (no speculative decoding):**
|
|
| Streams | Prompt Speed | Completion Speed | Total Time |
|
|
|---------|-------------|-----------------|------------|
|
|
| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
|
|
| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
|
|
| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
|
|
| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
|
|
|
|
Note: Speeds decrease with more streams due to KV cache contention.
|
|
|
|
**Still needs work:**
|
|
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
|
|
- Full context length support (32k is temporary workaround for OOM)
|
|
|
|
## Validation
|
|
|
|
Use `scripts/validate-vllm.sh` to test inference servers:
|
|
|
|
```bash
|
|
# Test dankserver (default)
|
|
bash scripts/validate-vllm.sh -t dankserver
|
|
|
|
# Test dankclaw
|
|
bash scripts/validate-vllm.sh -t dankclaw
|
|
|
|
# Custom host/port
|
|
bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model
|
|
|
|
# Run tests
|
|
bash tests/test_validator.sh dankserver
|
|
```
|