Frank
1949f0651b
- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo) - llama.cpp v10133 with ROCm0 offload, DFlash draft model - Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB) - Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB) - 3 parallel streams, 131072 context/stream - systemd service with minimal sandboxing (ROCm requires /dev/kfd access) - Baseline: 7.62 tok/s (threshold 5.0 tok/s)
vLLM Scripts
Scripts for running large language models via vLLM on two local machines.
Targets
| Host | User | Codename | Architecture |
|---|---|---|---|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
Models
| Model | Target | Format |
|---|---|---|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
| Qwen3.6 27B FP8 | dankclaw | FP8 |
Structure
docs/— Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notesscripts/— Executable vLLM launch scripts (coming soon)
Knowledge Base
See docs/README.md for the project knowledge base index.