dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup

Browse Source 
- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
Frank
2026-08-17 03:30:00 -07:00
parent 3f88b8e1ed
commit 968066ff23
13 changed files with 190 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+53 -26
View File
@@ -8,11 +8,43 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`.
The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).
## Boot Policy (2026-08-17)
Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.
Disabled units:
| Machine | Unit | Fate |
|---------|------|------|
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |
`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:
```
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>
```
The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.
The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:
```
sudo systemctl restart frank-pp-worker
```
`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.
## Workflow
@@ -77,7 +109,7 @@ Each target has a root-owned helper script that performs the full restart. The u
| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps:
@@ -87,7 +119,7 @@ The Dankclaw helper does these steps:
4. Reloads systemd.
5. Enables and restarts the service.
The Dankserver helper does the same steps without the cache drop.
The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.
The sudoers drop-in contains exactly two rules:
@@ -125,17 +157,24 @@ curl -sf http://127.0.0.1:7770/health
### Update Dankserver
Run these commands on Dankserver:
Run these commands on Dankserver to update and restart the PP worker:
```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
sudo systemctl restart frank-pp-worker
```
The worker has no HTTP health endpoint. Verify the listener instead:
```
ss -tln | grep :50052
```
The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.
## Service Units
### Dankclaw — frank-default-large
@@ -189,15 +228,9 @@ layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — frank-default-small (STOPPED 2026-08-16)
### Dankserver — old llama.cpp 35B service (retired 2026-08-16)
> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.
## Validation
@@ -230,17 +263,11 @@ If a service segfaults at load with no obvious cause, compare the exported env a
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:
A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
### VRAM and RAM growth under load
This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See [[dankserver-vram-growth]] for the full findings, the monitoring setup, and the mitigation options.
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.
## See Also