From e29071caa1f490f0fe5156dbb5328bf6dd653ff4 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 13 Aug 2026 20:42:57 -0700 Subject: [PATCH] docs: passwordless restart helpers, parallel-streams incident, MTP drafter findings - deployment.md: document the sudoers NOPASSWD restart helpers on both targets, the exact-argument matching quirk, and the 2026-08-13 'premature job release' investigation (server never preempts; aborts were client-side; Dankserver crashed twice under researcher load). - DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename --spec-type mtp -> draft-mtp in b9180. - AGENTS.md: operational quirks for the helpers, journal abort-vs-release signature, and the Dankserver cublasSgemm crash signature. --- AGENTS.md | 3 ++ docs/DFlash DSpark Speculative Decoding.md | 35 +++++++++++++++ docs/deployment.md | 52 +++++++++++++++++++--- 3 files changed, 85 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 53d17cc..99f3e44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,9 +83,12 @@ graph TD ### Operational quirks - `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '' | sudo -S `. +- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md). - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. - Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. +- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). +- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. ### Dankclaw memory model diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index 8e948f2..f75b968 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -110,6 +110,41 @@ llama-server --hf-repo \ --spec-draft-n-max 2 ``` +Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding. + +### MTP drafter availability for Qwen3.6-27B + +MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673). + +Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse: + +| File | Architecture | Tensors | MTP tensors | +|------|--------------|---------|-------------| +| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 | +| `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 | + +The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar: + +``` +llama-server --model \ + --spec-draft-model \ + --spec-type draft-mtp \ + --spec-draft-n-max 2 +``` + +Compatible MTP-only draft GGUFs on Hugging Face: + +| Repo | Notes | +|------|-------| +| `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` | +| `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` | +| `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) | +| `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) | + +Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls. + +On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]). + ## Draft Models DFlash draft model sizes for the models on these servers: diff --git a/docs/deployment.md b/docs/deployment.md index 3875231..09dba60 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -58,17 +58,49 @@ The script does these actions: 1. Verifies that the local working tree has no uncommitted changes. 2. Pushes the `main` branch to origin. 3. Pulls the repository on the target machine. -4. Installs or updates the systemd unit. -5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached. -6. Reloads systemd and restarts the service. -7. Waits for the health endpoint to respond (over ssh). +4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart). +5. Waits for the health endpoint to respond (over ssh). -Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the password from stdin: +Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin: ``` FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw ``` +## Passwordless Restart + +Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in. + +| Target | Helper | Sudoers drop-in | +|--------|--------|-----------------| +| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` | +| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` | + +The Dankclaw helper does these steps: + +1. Runs `sync`. +2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`. +3. Symlinks the systemd unit. +4. Reloads systemd. +5. Enables and restarts the service. + +The Dankserver helper does the same steps without the cache drop. + +The sudoers drop-in contains exactly two rules: + +``` +dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh +dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large +``` + +Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`. + +To test that the setup works, run this command: + +``` +sudo -n /usr/local/sbin/frank-restart-large.sh +``` + ### Update Dankclaw 1. Pull the repository. @@ -144,6 +176,16 @@ Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve sc If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. +## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13) + +The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged: + +1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest. +2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit). +3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined. + +Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line. + ## See Also - [AGENTS.md](../AGENTS.md) — the git workflow rules