Frank
39db415348
- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism - VLLM Index: quick reference with commands, endpoints, parameters - Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04 - Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1 - RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption - Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration - Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup - Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide - Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
27 lines
738 B
Markdown
27 lines
738 B
Markdown
# vLLM Scripts
|
|
|
|
Scripts for running large language models via vLLM on two local machines.
|
|
|
|
## Targets
|
|
|
|
| Host | User | Codename | Architecture |
|
|
|------|------|----------|-------------|
|
|
| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
|
|
| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
|
|
|
|
## Models
|
|
|
|
| Model | Target | Format |
|
|
|-------|--------|--------|
|
|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 |
|
|
| Qwen3.6 27B FP8 | dankclaw | FP8 |
|
|
|
|
## Structure
|
|
|
|
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
|
|
- `scripts/` — Executable vLLM launch scripts (coming soon)
|
|
|
|
## Knowledge Base
|
|
|
|
See [[docs/README.md]] for the project knowledge base index.
|