Frank
ced4c6f46e
- Model: Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (was AWQ 4bit) - Image: vllm/vllm-openai:nightly (was :latest) - Spec method: mtp with 2 tokens (was dflash with 12) - Max model len: 200700 (was 32768) - Max num seqs: 3 (was 1) - Max batched tokens: 3072 (was 4096) - GPU mem util: 0.97 (was 0.96) - CPU offload: 30GB (was 40GB) - Add --enable-prefix-caching - Add --kv-cache-dtype auto
vLLM Scripts
Scripts for running large language models via vLLM on two local machines.
Targets
| Host | User | Codename | Architecture |
|---|---|---|---|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
Models
| Model | Target | Format |
|---|---|---|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
Structure
docs/— Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notesscripts/— Executable vLLM launch scripts (coming soon)
Knowledge Base
See docs/README.md for the project knowledge base index.