61 lines
2.3 KiB
Markdown
61 lines
2.3 KiB
Markdown
# vLLM Scripts
|
|
|
|
Scripts for running large language models via vLLM on two local machines.
|
|
|
|
## Targets
|
|
|
|
| Host | User | Codename | Architecture |
|
|
|------|------|----------|-------------|
|
|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
|
|
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
|
|
|
|
## Models
|
|
|
|
| Model | Target | Format |
|
|
|-------|--------|--------|
|
|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
|
|
| Qwen3.6 27B FP8 | dankclaw | FP8 |
|
|
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
|
|
| Qwen3.6 35B A3B DFlash | dankserver | Draft model |
|
|
|
|
## Structure
|
|
|
|
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
|
|
- `scripts/` — Executable vLLM launch scripts
|
|
- `vllm-qwen36.sh` — FP8 inference script for dankserver
|
|
- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
|
|
- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)
|
|
|
|
## Knowledge Base
|
|
|
|
See [[docs/README.md]] for the project knowledge base index.
|
|
|
|
## Lessons Learned
|
|
|
|
### AWQ + DFlash on Ampere Hardware (2026-08-08)
|
|
|
|
**What worked:**
|
|
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
|
|
- AWQ Marlin kernel engaged automatically for weight conversion
|
|
- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
|
|
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
|
|
|
|
**What didn't work:**
|
|
- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
|
|
- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
|
|
- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
|
|
- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead
|
|
|
|
**Working configuration:**
|
|
- `--max-model-len 32768` (32k context)
|
|
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
|
|
- `--gpu-memory-utilization 0.95`
|
|
- `--performance-mode throughput`
|
|
- `--enable-chunked-prefill`
|
|
- `--max-num-batched-tokens 1536`
|
|
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
|
|
|
|
**Still needs work:**
|
|
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
|
|
- Full context length support (32k is temporary workaround for OOM)
|