Frank
903d54c389
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770 with MTP speculative decoding. Align Dankserver to the running vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs (gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4 experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
41 lines
1.6 KiB
Markdown
41 lines
1.6 KiB
Markdown
# vLLM Scripts
|
|
|
|
Scripts for running LLM inference on two local machines.
|
|
|
|
## Targets
|
|
|
|
| Host | User | Codename | Architecture |
|
|
|------|------|----------|-------------|
|
|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
|
|
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
|
|
|
|
## Active Services
|
|
|
|
| Service | Host | Runtime | Model | Port |
|
|
|---------|------|---------|-------|------|
|
|
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
|
|
| vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 |
|
|
|
|
> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. Its unit file `scripts/llama-server.service` remains for reference.
|
|
|
|
## Models
|
|
|
|
| Model | Target | Format |
|
|
|-------|--------|--------|
|
|
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
|
|
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ |
|
|
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
|
|
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
|
|
|
|
## Structure
|
|
|
|
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes
|
|
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-vllm-qwen36-dflash.sh, validate-vllm.sh, systemd unit files)
|
|
|
|
## Deployment
|
|
|
|
See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.
|
|
|
|
## Knowledge Base
|
|
|
|
See [docs/index.md](docs/index.md) for the project knowledge base index. |