vLLM Scripts

Scripts for running large language models via vLLM on two local machines.

Targets

Host	User	Codename	Architecture
dankserver	dank	DankServer	NVIDIA GPU + System RAM
dankclaw	dank	DankClaw	AMD Strix Halo (Shared Memory)

Models

Model	Target	Format
Qwen3.6 35B A3B FP8	dankserver	FP8
Qwen3.6 27B FP8	dankclaw	FP8
Qwen3.6 35B A3B AWQ 4bit	dankserver	AWQ 4bit
Qwen3.6 35B A3B DFlash	dankserver	Draft model

Structure

• docs/ — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
• scripts/ — Executable vLLM launch scripts
• vllm-qwen36.sh — FP8 inference script for dankserver
• vllm-awq-qwen36-dflash.sh — AWQ 4bit with DFlash speculative decoding (docker)
• vllm-awq-qwen36-dflash-ampere.sh — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)

Knowledge Base

See docs/README.md for the project knowledge base index.

Lessons Learned

AWQ + DFlash on Ampere Hardware (2026-08-08)

What worked:

• cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit loads successfully on Ampere GPUs
• AWQ Marlin kernel engaged automatically for weight conversion
• DFlash draft model z-lab/Qwen3.6-35B-A3B-DFlash detected as EAGLE model
• torch.compile with cudagraphs works with --max-num-batched-tokens 1536

What didn't work:

• --enable-prefix-caching conflicts with Mamba cache "align" mode — causes AssertionError in unify_kv_cache_spec_page_size
• --max-model-len 262144 causes OOM during AWQ→Marlin weight conversion on 24GB GPU
• Nightly vLLM --speculative-config requires num_speculative_tokens when model is provided
• --speculative-model flag removed in nightly; use --speculative-config JSON instead

Working configuration:

• --max-model-len 32768 (32k context)
• --cpu-offload-gb 40 with --cpu-offload-params "experts"
• --gpu-memory-utilization 0.95
• --performance-mode throughput
• --enable-chunked-prefill
• --max-num-batched-tokens 1536
• Thread binding: OMP_NUM_THREADS=8, VLLM_CPU_OMP_THREADS_BIND=8

Still needs work:

• DFlash speculative decoding integration (KV cache assertion bug in nightly)
• Full context length support (32k is temporary workaround for OOM)