docs: passwordless restart helpers, parallel-streams incident, MTP drafter findings
- deployment.md: document the sudoers NOPASSWD restart helpers on both targets, the exact-argument matching quirk, and the 2026-08-13 'premature job release' investigation (server never preempts; aborts were client-side; Dankserver crashed twice under researcher load). - DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename --spec-type mtp -> draft-mtp in b9180. - AGENTS.md: operational quirks for the helpers, journal abort-vs-release signature, and the Dankserver cublasSgemm crash signature.
This commit is contained in:
+47
-5
@@ -58,17 +58,49 @@ The script does these actions:
|
||||
1. Verifies that the local working tree has no uncommitted changes.
|
||||
2. Pushes the `main` branch to origin.
|
||||
3. Pulls the repository on the target machine.
|
||||
4. Installs or updates the systemd unit.
|
||||
5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached.
|
||||
6. Reloads systemd and restarts the service.
|
||||
7. Waits for the health endpoint to respond (over ssh).
|
||||
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
|
||||
5. Waits for the health endpoint to respond (over ssh).
|
||||
|
||||
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the password from stdin:
|
||||
Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:
|
||||
|
||||
```
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
|
||||
```
|
||||
|
||||
## Passwordless Restart
|
||||
|
||||
Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.
|
||||
|
||||
| Target | Helper | Sudoers drop-in |
|
||||
|--------|--------|-----------------|
|
||||
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
|
||||
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |
|
||||
|
||||
The Dankclaw helper does these steps:
|
||||
|
||||
1. Runs `sync`.
|
||||
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
|
||||
3. Symlinks the systemd unit.
|
||||
4. Reloads systemd.
|
||||
5. Enables and restarts the service.
|
||||
|
||||
The Dankserver helper does the same steps without the cache drop.
|
||||
|
||||
The sudoers drop-in contains exactly two rules:
|
||||
|
||||
```
|
||||
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
|
||||
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
|
||||
```
|
||||
|
||||
Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.
|
||||
|
||||
To test that the setup works, run this command:
|
||||
|
||||
```
|
||||
sudo -n /usr/local/sbin/frank-restart-large.sh
|
||||
```
|
||||
|
||||
### Update Dankclaw
|
||||
|
||||
1. Pull the repository.
|
||||
@@ -144,6 +176,16 @@ Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve sc
|
||||
|
||||
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
|
||||
|
||||
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
||||
|
||||
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:
|
||||
|
||||
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
|
||||
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
|
||||
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.
|
||||
|
||||
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
|
||||
|
||||
## See Also
|
||||
|
||||
- [AGENTS.md](../AGENTS.md) — the git workflow rules
|
||||
|
||||
Reference in New Issue
Block a user