dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fleet redo 2026-08-17: retire PP+NPU, new topology

Browse Source 
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
This commit is contained in:
Frank committed 2026-08-17 15:23:46 -07:00
1 parent 136ee080c9
commit d9af7f1d19
45 files changed
+961 -469
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/deployment.md
+64 -179
View File
@@ -8,43 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).
## Boot Policy (2026-08-17)
Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.
Disabled units:
| Machine | Unit | Fate |
|---------|------|------|
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |
`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:
```
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>
```
The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.
The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:
```
sudo systemctl restart frank-pp-worker
```
`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
## Workflow
@@ -70,22 +35,22 @@ graph LR
The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
Run this command to deploy to Dankclaw:
```
./scripts/deploy.sh dankclaw
```
Run this command to deploy to Dankserver:
```
./scripts/deploy.sh dankserver
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver
```
Run this command to deploy to Dankclaw:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```
Add the `--dry-run` flag to preview without making changes:
```
./scripts/deploy.sh dankclaw --dry-run
./scripts/deploy.sh dankserver --dry-run
```
The script does these actions:
@@ -93,153 +58,81 @@ The script does these actions:
1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
4. Symlinks the systemd unit, reloads systemd, enables, and restarts the service.
5. Waits for the health endpoint to respond (over ssh).
Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the
password from stdin (`sudo -S`). The passwordless restart helpers from the
retired topology are obsolete and are not used.
Health endpoints:
- Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:52625/health`
## Manual Update
### Update Dankserver (frank-default-large)
Run these commands on Dankserver:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```
## Passwordless Restart
Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.
| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps:
1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.
The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.
The sudoers drop-in contains exactly two rules:
```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```
Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.
To test that the setup works, run this command:
```
sudo -n /usr/local/sbin/frank-restart-large.sh
```
### Update Dankclaw
1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.
### Update Dankclaw (frank-default-small)
Run these commands on Dankclaw:
```
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:52625/health
```
### Update Dankserver
Run these commands on Dankserver to update and restart the PP worker:
```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload
sudo systemctl restart frank-pp-worker
```
The worker has no HTTP health endpoint. Verify the listener instead:
```
ss -tln | grep :50052
```
The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.
## Service Units
### Dankclaw — frank-default-large
### Dankserver — frank-default-large
The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti,
imatrix) under the alias `frank-default-large` on port 7770. The head
splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver
(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the
maximum for this config (measured 2026-08-17; see the PP doc).
Speculative decoding runs **MTP** with a standalone draft head
(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max
4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU
side runs **ROCm0** (no `--device` flag; default device selection puts
RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context
(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16**
(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with
Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full
layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
256, `--direct-io`. The unit file is
The active service on Dankserver is llama.cpp serving
`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias
`frank-default-large` on port 7770. It runs the patched FastMTP build
(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be +
the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type
draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the
ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16,
batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
`scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/serve-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`.
> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head,
> mismatched q8_0/f16 KV with FA (both binaries rebuilt with
> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the
> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance
> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.
### Dankclaw — frank-default-small
> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
> output; fixed by clearing the cache + freeing the disk. Guard:
> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
> production as `frank-default-large` (renamed from `frank-pp-head`);
> the standalone single-node service retired. MTP was re-enabled at
> `n_max 4` over the ngram-mod fallback. See
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
> configuration; the PP head runs ROCm0 by default (HIP registers before
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — old llama.cpp 35B service (retired 2026-08-16)
Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.
The active service on Dankclaw is vLLM (ROCm) serving
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
speculative decoding, `--reasoning-parser qwen3`. The unit file is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.
## Validation
The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.
Run this command on Dankserver to test vLLM throughput:
The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:
```
./scripts/validate-vllm.sh
curl -s http://127.0.0.1:7770/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}'
```
## Troubleshooting
@@ -248,32 +141,24 @@ If the health check fails after a deploy:
1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
4. Check the service status with `systemctl status <service>`.
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring.
On 2026-08-13 the old `frank-default-large` service crash-looped for hours on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache directory, used the path `1.0`, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.
Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged.
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.
Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
## See Also
- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
the sibling Podman app stack on the same hosts; its OpenWebUI consumes
the `:7770` endpoint, and its Gitea instance hosts this repository
the sibling Podman app stack on the same hosts