# vLLM Scripts

Scripts for running LLM inference on two local machines.

## Targets

| Host | User | Codename | Architecture |
|------|------|----------|-------------|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |

## Active Services

| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
| frank-default-small | dankserver | llama.cpp (DFlash) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 |

> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference.

## Models

| Model | Target | Format |
|-------|--------|--------|
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
| Qwen3.6 35B A3B uncensored Q6_K_P (DFlash draft) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |

## Structure

- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files)

## Deployment

See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.

## Knowledge Base

See [docs/index.md](docs/index.md) for the project knowledge base index.