# vLLM Scripts

Scripts for running large language models via vLLM on two local machines.

## Targets

| Host | User | Codename | Architecture |
|------|------|----------|-------------|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |

## Models

| Model | Target | Format |
|-------|--------|--------|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
| Qwen3.6 35B A3B DFlash | dankserver | Draft model |

## Structure

- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
- `scripts/` — Executable vLLM launch scripts
- `vllm-qwen36.sh` — FP8 inference script for dankserver
- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)

## Knowledge Base

See [[docs/README.md]] for the project knowledge base index.

## Lessons Learned

### AWQ + DFlash on Ampere Hardware (2026-08-08)

**What worked:**
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
- AWQ Marlin kernel engaged automatically for weight conversion
- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`

**What didn't work:**
- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead

**Working configuration:**
- `--max-model-len 32768` (32k context)
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
- `--gpu-memory-utilization 0.95`
- `--performance-mode throughput`
- `--enable-chunked-prefill`
- `--max-num-batched-tokens 1536`
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`

**Still needs work:**
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
- Full context length support (32k is temporary workaround for OOM)