docs: passwordless restart helpers, parallel-streams incident, MTP drafter findings

- deployment.md: document the sudoers NOPASSWD restart helpers on both
  targets, the exact-argument matching quirk, and the 2026-08-13
  'premature job release' investigation (server never preempts; aborts
  were client-side; Dankserver crashed twice under researcher load).
- DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the
  HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only
  draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename
  --spec-type mtp -> draft-mtp in b9180.
- AGENTS.md: operational quirks for the helpers, journal abort-vs-release
  signature, and the Dankserver cublasSgemm crash signature.
This commit is contained in:
Frank
2026-08-13 20:42:57 -07:00
parent 6fae3a9f1a
commit e29071caa1
3 changed files with 85 additions and 5 deletions
+47 -5
View File
@@ -58,17 +58,49 @@ The script does these actions:
1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Installs or updates the systemd unit.
5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached.
6. Reloads systemd and restarts the service.
7. Waits for the health endpoint to respond (over ssh).
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Waits for the health endpoint to respond (over ssh).
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the password from stdin:
Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```
## Passwordless Restart
Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.
| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps:
1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.
The Dankserver helper does the same steps without the cache drop.
The sudoers drop-in contains exactly two rules:
```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```
Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.
To test that the setup works, run this command:
```
sudo -n /usr/local/sbin/frank-restart-large.sh
```
### Update Dankclaw
1. Pull the repository.
@@ -144,6 +176,16 @@ Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve sc
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
## See Also
- [AGENTS.md](../AGENTS.md) — the git workflow rules