Frank
e4fba35e89
- vLLM 0.26.0+rocm723 - --kv-cache-dtype fp8 --gpu-memory-utilization 0.80 - --max-model-len 32768 --max-num-batched-tokens 2048 - --max-num-seqs 32 --reasoning-parser qwen3 --language-model-only - Baseline response time: 219s for 50 tokens
vLLM Scripts
Scripts for running large language models via vLLM on two local machines.
Targets
| Host | User | Codename | Architecture |
|---|---|---|---|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
Models
| Model | Target | Format |
|---|---|---|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
Structure
docs/— Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notesscripts/— Executable vLLM launch scripts (coming soon)
Knowledge Base
See docs/README.md for the project knowledge base index.