Frank
968066ff23
- deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed
47 lines
1.8 KiB
Markdown
47 lines
1.8 KiB
Markdown
# vLLM Scripts
|
|
|
|
Deployment scripts for the LLM inference fleet: a llama.cpp
|
|
pipeline-parallel Qwen3.8-27B split across two machines, plus the
|
|
FastFlowLM NPU endpoint on Dankclaw.
|
|
|
|
## Targets
|
|
|
|
| Host | User | Codename | Architecture |
|
|
|------|------|----------|--------------|
|
|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
|
|
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
|
|
|
|
## Active Services
|
|
|
|
| Service | Host | Runtime | Model | Port |
|
|
|---------|------|---------|-------|------|
|
|
| frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 |
|
|
| frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 |
|
|
|
|
Boot policy (2026-08-17): only these two units are enabled. The
|
|
FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625)
|
|
is disabled for boot; re-enable it from `scripts/frank-default-small.service`
|
|
if needed. Retired units: the Dankserver llama.cpp 35B service,
|
|
`llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`,
|
|
`frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit
|
|
files remain in `scripts/` for reference.
|
|
|
|
## Structure
|
|
|
|
- `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix,
|
|
NPU notes, research archive
|
|
- `scripts/` — serve scripts, env files, systemd units, deploy and test
|
|
tools
|
|
|
|
## Deployment
|
|
|
|
See [docs/deployment.md](docs/deployment.md). Run
|
|
`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target
|
|
is broken (it targets the removed `frank-default-small` unit); restart
|
|
the worker with `sudo systemctl restart frank-pp-worker` until the
|
|
script is fixed.
|
|
|
|
## Knowledge Base
|
|
|
|
See [docs/index.md](docs/index.md) for the project knowledge base index.
|