dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fleet redo 2026-08-17: retire PP+NPU, new topology

Browse Source 
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
This commit is contained in:
Frank committed 2026-08-17 15:23:46 -07:00
1 parent 136ee080c9
commit d9af7f1d19
45 files changed
+961 -469
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+77 -55
View File
@@ -2,48 +2,44 @@
## Overview
This project manages deployment scripts for the LLM inference fleet.
Dankclaw runs a llama.cpp pipeline-parallel head serving
Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone
`-draft-Q8_0` sidecar, `--spec-draft-n-max 4`) over the `--spec-default`
ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors
split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 `ggml-rpc`
worker, KV q8_0/f16 (mismatched, FA retained via the
GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no `--device`
flag), port 7770.
The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
`qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is
stopped). **Boot policy (2026-08-17):** only `frank-default-large` and
`frank-pp-worker` are enabled for boot. The NPU unit `frank-default-small`
is disabled; the running service keeps serving until stopped or rebooted.
See [docs/deployment.md](docs/deployment.md) for the deployment
process, the boot policy, and the re-enable recipes.
This project manages deployment scripts for the LLM inference fleet on two
local machines. Topology (2026-08-17 redo):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
(no RPC worker). Dankclaw runs vLLM on the Strix Halo iGPU (gfx1151).
The old scripts live in `old/scripts/` for reference; they describe the
retired PP topology.
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
OpenWebUI consumes the `:7770` endpoint; its Gitea instance hosts this
repository. Cross-links live in both knowledge-base indexes.
OpenWebUI consumes the `:7770` endpoint on Dankserver; its Gitea instance
hosts this repository.
## Benchmarking Policy (MANDATORY)
**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service
when the bench finishes. The users will call if they need the model.
when the bench finishes.
Rules:
1. Stop the service before any benchmark: `systemctl stop frank-default-large`
(Dankclaw) or `systemctl stop frank-default-small` (Dankclaw NPU). Use the
passwordless helper or the password `Assface.3241` with `sudo -S`.
2. Run the bench on a scratch port (never the service port 7770).
3. Restart the service when the bench completes: `systemctl start <svc>`.
4. Verify health before reporting: `curl http://127.0.0.1:7770/health`.
Do not skip this policy. A stale GPU contention or a dead production box is
worse than a slow bench.
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-large` (Dankserver) or
`sudo systemctl stop frank-default-small` (Dankclaw). Use the password
`Assface.3241` with `sudo -S` when no tty is available.
2. Run the bench on a scratch port (never the service ports 7770/52625).
3. Restart the service when the bench completes:
`sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:52625/health`.
## Git Workflow
@@ -63,11 +59,15 @@ All script edits must flow through git. Do not edit files on the remote machine
Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.
```
./scripts/deploy.sh dankclaw # PP head (frank-default-large, port 7770)
./scripts/deploy.sh dankserver # PP worker (frank-pp-worker, RPC 50052)
./scripts/deploy.sh dankclaw --dry-run # preview without changes
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # vLLM, port 52625
./scripts/deploy.sh dankserver --dry-run # preview without changes
```
Set `FRANK_SUDO_PW` for headless runs (remote sudo reads it from stdin).
The passwordless restart helpers installed for the retired topology are
obsolete; `deploy.sh` no longer uses them.
Deployment workflow:
```mermaid
@@ -79,25 +79,45 @@ graph LR
E --> F[health check]
```
Target map:
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68/0.32 to the Dankserver worker via RPC, standalone MTP draft + ngram-mod, q8_0/f16 KV) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) — disabled for boot since 2026-08-17 | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 |
> Warning: `deploy.sh dankserver` is currently broken. It targets the
> removed `frank-default-small` unit. Restart the worker with
> `sudo systemctl restart frank-pp-worker` until the script is fixed.
## Runtime & Troubleshooting
### FastMTP patched llama.cpp (Dankserver)
The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch. See the
HauhauCS HF repo README for the exact build recipe.
### vLLM ROCm install (Dankclaw)
vLLM runs from `/home/dank/vllm-venv` (Python 3.12). Install recipe:
```
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
```
`--torch-backend=auto` resolves the AMD ROCm wheel set (torch +
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
The model cache is the default HF cache
(`/home/dank/.cache/huggingface`); the HF token lives at
`~/.cache/huggingface/token`. The retired CIRU package
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
### Env var naming rule — HIP runtime collision
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with `set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every model load. Commit `227ce01` fixed it by renaming the sampling variable to `TEMPERATURE`.
The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13
`TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then
segfaulted every model load. Commit `227ce01` fixed it by renaming the
sampling variable to `TEMPERATURE`.
Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for collisions with this command:
Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
collisions with this command:
```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
@@ -121,25 +141,27 @@ graph TD
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command:
```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-pp-head.sh
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
```
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
### Operational quirks
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Passwordless restarts: each target has a root-owned helper and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). The Dankclaw helper `/usr/local/sbin/frank-restart-large.sh` is correct for the PP head. The Dankserver helper `frank-restart-small.sh` is **obsolete** — it restarts the removed `frank-default-small` unit. Restart the worker with `sudo systemctl restart frank-pp-worker`. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw vLLM ~5-10 minutes (38 GB checkpoint). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature (historical): the retired 35B service aborted under parallel load with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes (llama.cpp b10121, driver 595.84). Root cause was never determined; the service is retired. Do not confuse that crash with speculative decoding. See [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md).
- DFlash premature EOS under parallel load (historical): on Dankclaw, `draft-dflash` ended a generating slot ~1s after a new task landed with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log — a DFlash shared-draft-context bug. DFlash and DSpark are retired from Dankclaw. Production uses the standalone MTP draft over the ngram-mod fallback at PARALLEL=1 on ROCm0. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
### Dankclaw memory model
The GPU budget on Dankclaw is the rocm-smi figure: 96 GiB addressable VRAM (SVM). The GPU node reports `local_mem_size 0`, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or `dmidecode` for the GPU budget; they show the CPU-side carve-out. This is intentional. Do not change it.
The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported
by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable
SVM pool). The GPU node reports `local_mem_size 0`, so all allocations are
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
`dmidecode` for the GPU budget; they show the CPU-side carve-out. This is
intentional. Do not change it.
## Why