104 lines
3.0 KiB
Markdown
Executable File
104 lines
3.0 KiB
Markdown
Executable File
# vLLM Scripts — Agent Instructions
|
|
|
|
## Project Overview
|
|
Scripts for running large language models via vLLM on two local machines:
|
|
- **dankserver**: NVIDIA GPU + System RAM (frank-default-small)
|
|
- **dankclaw**: AMD Strix Halo Shared Memory (frank-default-large)
|
|
|
|
## Directory Structure
|
|
```
|
|
vllmScripts/
|
|
├── AGENTS.md # This file — agent instructions
|
|
├── README.md # Human-readable project overview
|
|
├── deploy.sh # Automated deployment script
|
|
├── scripts/ # Executable vLLM launch scripts
|
|
│ ├── validate-vllm.sh # Validation tool
|
|
│ └── *.sh # Inference server scripts
|
|
├── tests/ # Test suite
|
|
│ └── test_validator.sh
|
|
├── docs/ # Knowledge base
|
|
│ ├── README.md
|
|
│ └── *.md # Research notes
|
|
└── failedAttempts/ # Archived scripts (for reference)
|
|
```
|
|
|
|
## Running Tests
|
|
```bash
|
|
# Test validator on dankserver
|
|
bash tests/test_validator.sh dankserver
|
|
|
|
# Test validator on dankclaw
|
|
bash tests/test_validator.sh dankclaw
|
|
|
|
# Run all tests
|
|
bash tests/test_validator.sh dankserver && bash tests/test_validator.sh dankclaw
|
|
```
|
|
|
|
## Deployment
|
|
```bash
|
|
# Deploy all scripts to all targets
|
|
bash deploy.sh all
|
|
|
|
# Deploy specific script to dankserver
|
|
bash deploy.sh dankserver vllm-qwen36.sh
|
|
|
|
# Deploy specific script to dankclaw
|
|
bash deploy.sh dankclaw vllm-thinkingcap.sh
|
|
```
|
|
|
|
## Git Workflow
|
|
|
|
### Branching Strategy
|
|
- `main`: Stable, working scripts only
|
|
- `dankserver/*`: Server-specific development
|
|
- `dankserver/qwen36-awq`: Qwen3.6 AWQ experiments
|
|
- `dankserver/*`: Feature branches for dankserver
|
|
- `dankclaw/*`: Claw-specific development
|
|
- `dankclaw/*`: Feature branches for dankclaw
|
|
|
|
### Worktree Pattern
|
|
For parallel development:
|
|
```bash
|
|
# Create worktree for feature
|
|
git worktree add ../vllm-feature feature/qwen36-awq
|
|
|
|
# When done, merge back
|
|
git worktree remove ../vllm-feature
|
|
```
|
|
|
|
### Commit Guidelines
|
|
- Use descriptive commit messages: `git commit -m "Add AWQ DFlash script"`
|
|
- Commit before deploying: `git add -A && git commit -m "Update for deployment"`
|
|
- Push to remote: `git push origin <branch>`
|
|
|
|
### Deployment Flow
|
|
1. Make changes locally
|
|
2. Run tests: `bash tests/test_validator.sh dankserver`
|
|
3. Commit: `git add -A && git commit -m "Description"`
|
|
4. Deploy: `bash deploy.sh dankserver`
|
|
5. Verify: `bash scripts/validate-vllm.sh -t dankserver`
|
|
|
|
## Remote Servers
|
|
|
|
### dankserver
|
|
- SSH: `dank@dankserver`
|
|
- Scripts: `~/vllm/`
|
|
- Service: `vllm-awq-qwen36.service`
|
|
- Model: frank-default-small (AWQ 4bit)
|
|
- Port: 34401
|
|
- Status: Disabled (pending bug fixes)
|
|
|
|
### dankclaw
|
|
- SSH: `dank@dankclaw`
|
|
- Scripts: `~/vllm/`
|
|
- Service: `vllm-thinkingcap.service`
|
|
- Model: frank-default-large (FP8)
|
|
- Port: 3090
|
|
- Status: Running (you are the deployed inference)
|
|
|
|
## Troubleshooting
|
|
- Service failed: `systemctl status <service>`
|
|
- Container stuck: `docker stop <name> && docker rm <name>`
|
|
- OOM error: Reduce `--max-model-len` or increase `--cpu-offload-gb`
|
|
- Prefix caching conflict: Remove `--enable-prefix-caching` from script
|