# vLLM Scripts

Scripts for running large language models via vLLM on two local machines.

## Targets

| Host | User | Codename | Architecture | Service Status |
|------|------|----------|-------------|----------------|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM | `vllm-awq-qwen36.service` (failed) |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) | `vllm-thinkingcap.service` (failed) |

## File Locations

### Local (vllmScripts/)
- `scripts/` — Active inference scripts
- `tests/` — Test suite
- `docs/` — Knowledge base
- `failedAttempts/` — Archived scripts

### Remote (dankserver)
- `~/vllm/` — Deployed inference scripts
- `~/vllm/failedAttempts/` — Archived scripts
- `~/vLLM/` — Legacy project directory (vllm-qwen36.sh)

### Remote (dankclaw)
- `~/vllm/` — vLLM source code (not inference scripts)
- `~/vLLM/` — Inference scripts (if deployed)

## Models

| Model | Target | Format |
|-------|--------|--------|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
| Qwen3.6 35B A3B DFlash | dankserver | Draft model |

## Structure

- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
- `scripts/` — Executable vLLM launch scripts
- `vllm-qwen36.sh` — FP8 inference script for dankserver
- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)

## Knowledge Base

See [[docs/README.md]] for the project knowledge base index.

## Lessons Learned

### AWQ + DFlash on Ampere Hardware (2026-08-08)

**What worked:**
- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
- AWQ Marlin kernel engaged automatically for weight conversion
- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`

**What didn't work:**
- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead

**Working configuration:**
- `--max-model-len 32768` (32k context)
- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
- `--gpu-memory-utilization 0.95`
- `--performance-mode throughput`
- `--enable-chunked-prefill`
- `--max-num-batched-tokens 1536`
- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`

**Performance observations (no speculative decoding):**
| Streams | Prompt Speed | Completion Speed | Total Time |
|---------|-------------|-----------------|------------|
| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |

Note: Speeds decrease with more streams due to KV cache contention.

**Still needs work:**
- DFlash speculative decoding integration (KV cache assertion bug in nightly)
- Full context length support (32k is temporary workaround for OOM)

## Validation

Use `scripts/validate-vllm.sh` to test inference servers:

```bash
# Test dankserver (default)
bash scripts/validate-vllm.sh -t dankserver

# Test dankclaw
bash scripts/validate-vllm.sh -t dankclaw

# Custom host/port
bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model

# Run tests
bash tests/test_validator.sh dankserver
```