Host	User	Codename	Architecture	Service Status
dankserver	dank	DankServer	NVIDIA GPU + System RAM	`vllm-awq-qwen36.service` (failed)
dankclaw	dank	DankClaw	AMD Strix Halo (Shared Memory)	`vllm-thinkingcap.service` (failed)

File Locations

Local (vllmScripts/)

scripts/ — Active inference scripts
tests/ — Test suite
docs/ — Knowledge base
failedAttempts/ — Archived scripts

Remote (dankserver)

~/vllm/ — Deployed inference scripts
~/vllm/failedAttempts/ — Archived scripts
~/vLLM/ — Legacy project directory (vllm-qwen36.sh)

Remote (dankclaw)

~/vllm/ — vLLM source code (not inference scripts)
~/vLLM/ — Inference scripts (if deployed)

Models

Model	Target	Format
Qwen3.6 35B A3B FP8	dankserver	FP8
Qwen3.6 27B FP8	dankclaw	FP8
Qwen3.6 35B A3B AWQ 4bit	dankserver	AWQ 4bit
Qwen3.6 35B A3B DFlash	dankserver	Draft model

Structure

docs/ — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
scripts/ — Executable vLLM launch scripts
vllm-qwen36.sh — FP8 inference script for dankserver
vllm-awq-qwen36-dflash.sh — AWQ 4bit with DFlash speculative decoding (docker)
vllm-awq-qwen36-dflash-ampere.sh — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)

Knowledge Base

See docs/README.md for the project knowledge base index.

Lessons Learned

AWQ + DFlash on Ampere Hardware (2026-08-08)

What worked:

cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit loads successfully on Ampere GPUs
AWQ Marlin kernel engaged automatically for weight conversion
DFlash draft model z-lab/Qwen3.6-35B-A3B-DFlash detected as EAGLE model
torch.compile with cudagraphs works with --max-num-batched-tokens 1536

What didn't work:

--enable-prefix-caching conflicts with Mamba cache "align" mode — causes AssertionError in unify_kv_cache_spec_page_size
--max-model-len 262144 causes OOM during AWQ→Marlin weight conversion on 24GB GPU
Nightly vLLM --speculative-config requires num_speculative_tokens when model is provided
--speculative-model flag removed in nightly; use --speculative-config JSON instead

Working configuration:

--max-model-len 32768 (32k context)
--cpu-offload-gb 40 with --cpu-offload-params "experts"
--gpu-memory-utilization 0.95
--performance-mode throughput
--enable-chunked-prefill
--max-num-batched-tokens 1536
Thread binding: OMP_NUM_THREADS=8, VLLM_CPU_OMP_THREADS_BIND=8

Performance observations (no speculative decoding):

Streams	Prompt Speed	Completion Speed	Total Time
1	4.27 tok/s	21.42 tok/s	11,951ms
2	2.44 tok/s	12.25 tok/s	20,891ms
3	1.35 tok/s	6.76 tok/s	37,849ms
4	1.43 tok/s	7.19 tok/s	35,606ms

Note: Speeds decrease with more streams due to KV cache contention.

Still needs work:

DFlash speculative decoding integration (KV cache assertion bug in nightly)
Full context length support (32k is temporary workaround for OOM)

Validation

Use scripts/validate-vllm.sh to test inference servers:

# Test dankserver (default)
bash scripts/validate-vllm.sh -t dankserver

# Test dankclaw
bash scripts/validate-vllm.sh -t dankclaw

# Custom host/port
bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model

# Run tests
bash tests/test_validator.sh dankserver