# AGENTS.md — vllmScripts Agent Rules

## Overview

This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug).

## Benchmarking Policy (MANDATORY)

**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service
when the bench finishes. The users will call if they need the model.

Rules:

1. Stop the service before any benchmark: `systemctl stop frank-default-large`
   (Dankclaw) or `systemctl stop frank-default-small` (Dankserver). Use the
   passwordless helper or the password `Assface.3241` with `sudo -S`.
2. Run the bench on a scratch port (never the service port 7770).
3. Restart the service when the bench completes: `systemctl start <svc>`.
4. Verify health before reporting: `curl http://127.0.0.1:7770/health`.

Do not skip this policy. A stale GPU contention or a dead production box is
worse than a slow bench.

## Git Workflow

All script edits must flow through git. Do not edit files on the remote machine directly.

### Protocol

1. **Source the script from the target machine** — fetch it from the server where it runs
2. **Place it in this repo** — commit the fetched version
3. **Edit here** — make all changes locally in the working tree
4. **Commit and push** — push to origin (Gitea on dankclaw)
5. **Pull on the target** — `git pull` on the remote machine before restarting
6. **Restart the service** — use the updated script on the target

## Deployment & Update Process

Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.

```
./scripts/deploy.sh dankclaw        # vLLM (frank-default-large, port 7770)
./scripts/deploy.sh dankserver      # llama.cpp (frank-default-small, port 7770)
./scripts/deploy.sh dankclaw --dry-run   # preview without changes
```

Deployment workflow:

```mermaid
graph LR
    A[Edit locally] --> B[Commit]
    B --> C[Push to origin]
    C --> D[deploy.sh: pull on target]
    D --> E[restart service]
    E --> F[health check]
```

Target map:

| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 |
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |

## Runtime & Troubleshooting

### Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with `set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every model load. Commit `227ce01` fixed it by renaming the sampling variable to `TEMPERATURE`.

Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for collisions with this command:

```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
```

### Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

```mermaid
graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]
```

1. Read the service journal with `journalctl -u <service> -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command:
   ```
   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-llama-frank-large.sh
   ```
   `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.

### Operational quirks

- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md).

### Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 96 GiB addressable VRAM (SVM). The GPU node reports `local_mem_size 0`, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or `dmidecode` for the GPU budget; they show the CPU-side carve-out. This is intentional. Do not change it.

## Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

> User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart

## Summary

| Context | Action |
|---------|--------|
| Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. |
| Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |