docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup
- deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
1 parent
3f88b8e1ed
commit
968066ff23
13 files changed
+190
-246
No files matched your search
@@ -1,11 +1,13 @@
|
||||
# vLLM Scripts
|
||||
|
||||
Scripts for running LLM inference on two local machines.
|
||||
Deployment scripts for the LLM inference fleet: a llama.cpp
|
||||
pipeline-parallel Qwen3.8-27B split across two machines, plus the
|
||||
FastFlowLM NPU endpoint on Dankclaw.
|
||||
|
||||
## Targets
|
||||
|
||||
| Host | User | Codename | Architecture |
|
||||
|------|------|----------|-------------|
|
||||
|------|------|----------|--------------|
|
||||
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
|
||||
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
|
||||
|
||||
@@ -13,30 +15,32 @@ Scripts for running LLM inference on two local machines.
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
|
||||
| frank-default-small | dankserver | llama.cpp (ngram-mod) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 |
|
||||
| frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 |
|
||||
| frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 |
|
||||
|
||||
> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference.
|
||||
|
||||
## Models
|
||||
|
||||
| Model | Target | Format |
|
||||
|-------|--------|--------|
|
||||
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
|
||||
| Qwen3.6 35B A3B uncensored Q6_K_P (ngram-mod speculative decoding) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) |
|
||||
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) |
|
||||
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
|
||||
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
|
||||
Boot policy (2026-08-17): only these two units are enabled. The
|
||||
FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625)
|
||||
is disabled for boot; re-enable it from `scripts/frank-default-small.service`
|
||||
if needed. Retired units: the Dankserver llama.cpp 35B service,
|
||||
`llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`,
|
||||
`frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit
|
||||
files remain in `scripts/` for reference.
|
||||
|
||||
## Structure
|
||||
|
||||
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes
|
||||
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files)
|
||||
- `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix,
|
||||
NPU notes, research archive
|
||||
- `scripts/` — serve scripts, env files, systemd units, deploy and test
|
||||
tools
|
||||
|
||||
## Deployment
|
||||
|
||||
See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.
|
||||
See [docs/deployment.md](docs/deployment.md). Run
|
||||
`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target
|
||||
is broken (it targets the removed `frank-default-small` unit); restart
|
||||
the worker with `sudo systemctl restart frank-pp-worker` until the
|
||||
script is fixed.
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
See [docs/index.md](docs/index.md) for the project knowledge base index.
|
||||
See [docs/index.md](docs/index.md) for the project knowledge base index.
|
||||
Reference in new issue
Block a user