dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fleet redo 2026-08-17: retire PP+NPU, new topology

Browse Source 
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
This commit is contained in:
Frank committed 2026-08-17 15:23:46 -07:00
1 parent 136ee080c9
commit d9af7f1d19
45 files changed
+961 -469
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+16 -24
View File
@@ -1,45 +1,37 @@
# vLLM Scripts
Deployment scripts for the LLM inference fleet: a llama.cpp
pipeline-parallel Qwen3.8-27B split across two machines, plus the
FastFlowLM NPU endpoint on Dankclaw.
Deployment scripts for the LLM inference fleet (2026-08-17 topology):
llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.
## Targets
| Host | User | Codename | Architecture |
|------|------|----------|--------------|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
| Host | User | Codename | GPU |
|------|------|----------|-----|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 24 GB + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo iGPU gfx1151 (unified memory) |
## Active Services
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 |
| frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 |
| frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 |
| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quantization, 4 seqs, ngram spec | 52625 |
Boot policy (2026-08-17): only these two units are enabled. The
FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625)
is disabled for boot; re-enable it from `scripts/frank-default-small.service`
if needed. Retired units: the Dankserver llama.cpp 35B service,
`llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`,
`frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit
files remain in `scripts/` for reference.
The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in
`old/scripts/` for reference.
## Structure
- `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix,
NPU notes, research archive
- `scripts/` — serve scripts, env files, systemd units, deploy and test
tools
- `docs/` — knowledge base: deployment runbook, model notes, research archive
- `scripts/` — serve scripts, env files, systemd units, deploy and test tools
- `old/scripts/` — retired PP/NPU-era scripts (reference only)
## Deployment
See [docs/deployment.md](docs/deployment.md). Run
`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target
is broken (it targets the removed `frank-default-small` unit); restart
the worker with `sudo systemctl restart frank-pp-worker` until the
script is fixed.
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver` for the llama.cpp box or
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the vLLM box.
## Knowledge Base