dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: passwordless restart helpers, parallel-streams incident, MTP drafter findings

Browse Source 
- deployment.md: document the sudoers NOPASSWD restart helpers on both
  targets, the exact-argument matching quirk, and the 2026-08-13
  'premature job release' investigation (server never preempts; aborts
  were client-side; Dankserver crashed twice under researcher load).
- DFlash/DSpark doc: MTP requires MTP tensors in the target GGUF; the
  HauhauCS Qwen3.6-27B GGUF has none (verified tensor listing); MTP-only
  draft GGUFs (a4lg/lym00) pair via --spec-draft-model; flag rename
  --spec-type mtp -> draft-mtp in b9180.
- AGENTS.md: operational quirks for the helpers, journal abort-vs-release
  signature, and the Dankserver cublasSgemm crash signature.
This commit is contained in:
Frank
2026-08-13 20:42:57 -07:00
parent 6fae3a9f1a
commit e29071caa1
3 changed files with 85 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+3
View File
@@ -83,9 +83,12 @@ graph TD
### Operational quirks ### Operational quirks
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`. - `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. - Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process.
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
### Dankclaw memory model ### Dankclaw memory model
docs/DFlash DSpark Speculative Decoding.md
+35
View File
@@ -110,6 +110,41 @@ llama-server --hf-repo <target-with-MTP> \
--spec-draft-n-max 2 --spec-draft-n-max 2
``` ```
Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.
### MTP drafter availability for Qwen3.6-27B
MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673).
Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:
| File | Architecture | Tensors | MTP tensors |
|------|--------------|---------|-------------|
| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 |
| `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 |
The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:
```
llama-server --model <HauhauCS-Q8_K_P.gguf> \
--spec-draft-model <MTP-only-draft.gguf> \
--spec-type draft-mtp \
--spec-draft-n-max 2
```
Compatible MTP-only draft GGUFs on Hugging Face:
| Repo | Notes |
|------|-------|
| `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` |
| `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` |
| `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) |
| `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) |
Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.
On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
## Draft Models ## Draft Models
DFlash draft model sizes for the models on these servers: DFlash draft model sizes for the models on these servers:
docs/deployment.md
+47 -5
View File
@@ -58,17 +58,49 @@ The script does these actions:
1. Verifies that the local working tree has no uncommitted changes. 1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin. 2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine. 3. Pulls the repository on the target machine.
4. Installs or updates the systemd unit. 4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached. 5. Waits for the health endpoint to respond (over ssh).
6. Reloads systemd and restarts the service.
7. Waits for the health endpoint to respond (over ssh).
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the password from stdin: Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:
``` ```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
``` ```
## Passwordless Restart
Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.
| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps:
1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.
The Dankserver helper does the same steps without the cache drop.
The sudoers drop-in contains exactly two rules:
```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```
Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.
To test that the setup works, run this command:
```
sudo -n /usr/local/sbin/frank-restart-large.sh
```
### Update Dankclaw ### Update Dankclaw
1. Pull the repository. 1. Pull the repository.
@@ -144,6 +176,16 @@ Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve sc
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
## See Also ## See Also
- [AGENTS.md](../AGENTS.md) — the git workflow rules - [AGENTS.md](../AGENTS.md) — the git workflow rules