Frank
4a17948443
- serve-ling3-vllm.sh: wrapper around jcbtc validated run-256k.sh (TRITON_MLA, MTP K1, 256k, 6 slots, ROCr idle-fix LD_PRELOAD); sets HOST/PORT/ SERVED_MODEL_NAME=frank-default-large - frank-default-large.service: systemd unit, User=dank, port 7770
vLLM Scripts
Scripts for running LLM inference on two local machines.
Targets
| Host | User | Codename | Architecture |
|---|---|---|---|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
Active Services
| Service | Host | Runtime | Model | Port |
|---|---|---|---|---|
| llama-server | dankclaw | ROCmFPX llama.cpp | Ling-3.0-flash ROCmFP4 MTP | 8080 |
| vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 |
Models
| Model | Target | Format |
|---|---|---|
| Ling-3.0-flash ROCmFP4-MTP | dankclaw | ROCmFP4 GGUF |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
Structure
docs/— Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notesscripts/— Deployment and launch scripts (deploy.sh, serve-vllm-qwen36-dflash.sh, validate-vllm.sh, systemd unit files)
Deployment
See docs/deployment.md for the deploy/update process. Run ./scripts/deploy.sh <dankclaw|dankserver> after pushing committed changes.
Knowledge Base
See docs/index.md for the project knowledge base index.