serverAppConfigs/AGENTS.md

# AGENTS.md — vllmScripts Agent Rules

## Overview

This project manages deployment scripts for the LLM inference fleet on two
local machines. Topology (2026-08-17 redo):

| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec | 52625 |

The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
(no RPC worker). Dankclaw runs vLLM on the Strix Halo iGPU (gfx1151).
The old scripts live in `old/scripts/` for reference; they describe the
retired PP topology.

**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
OpenWebUI consumes the `:7770` endpoint on Dankserver; its Gitea instance
hosts this repository.

## Benchmarking Policy (MANDATORY)

**STOP the production service BEFORE running any benchmark on a target.**
These servers serve real users (about 3 people). Running a bench while the
service is up starves the GPU and corrupts results. Restart the service
when the bench finishes.

Rules:

1. Stop the service before any benchmark:
   `sudo systemctl stop frank-default-large` (Dankserver) or
   `sudo systemctl stop frank-default-small` (Dankclaw). Use the password
   `Assface.3241` with `sudo -S` when no tty is available.
2. Run the bench on a scratch port (never the service ports 7770/52625).
3. Restart the service when the bench completes:
   `sudo systemctl start <svc>`.
4. Verify health before reporting:
   `curl http://192.168.0.235:7770/health` or
   `curl http://192.168.0.143:52625/health`.

## Git Workflow

All script edits must flow through git. Do not edit files on the remote machine directly.

### Protocol

1. **Source the script from the target machine** — fetch it from the server where it runs
2. **Place it in this repo** — commit the fetched version
3. **Edit here** — make all changes locally in the working tree
4. **Commit and push** — push to origin (Gitea on dankclaw)
5. **Pull on the target** — `git pull` on the remote machine before restarting
6. **Restart the service** — use the updated script on the target

## Deployment & Update Process

Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.

```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver   # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw     # vLLM, port 52625
./scripts/deploy.sh dankserver --dry-run           # preview without changes
```

Set `FRANK_SUDO_PW` for headless runs (remote sudo reads it from stdin).
The passwordless restart helpers installed for the retired topology are
obsolete; `deploy.sh` no longer uses them.

Deployment workflow:

```mermaid
graph LR
    A[Edit locally] --> B[Commit]
    B --> C[Push to origin]
    C --> D[deploy.sh: pull on target]
    D --> E[restart service]
    E --> F[health check]
```

## Runtime & Troubleshooting

### FastMTP patched llama.cpp (Dankserver)

The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
rebuild from another commit without re-applying the patch.

**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe.

### vLLM ROCm install (Dankclaw)

vLLM runs from `/home/dank/vllm-venv` (Python 3.12). Install recipe:

```
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
```

`--torch-backend=auto` resolves the AMD ROCm wheel set (torch +
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
The model cache is the default HF cache
(`/home/dank/.cache/huggingface`); the HF token lives at
`~/.cache/huggingface/token`. The retired CIRU package
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.

**ROCm flash-attention requirement:** the vision encoder needs the Triton
flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
`frank-default-small.env`) and keep the pure-Python `flash_attn` package
(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).

### Env var naming rule — HIP runtime collision

The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13
`TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then
segfaulted every model load. Commit `227ce01` fixed it by renaming the
sampling variable to `TEMPERATURE`.

Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
collisions with this command:

```
strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
```

### Crash-loop diagnostic procedure

When a llama service segfaults during model load, follow these steps.

```mermaid
graph TD
    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
    B --> C{Segfault in libamdhip64?}
    C -- yes --> D[Bisect script vs raw binary]
    C -- no --> E[Check config, memory, model files]
    D --> F[Run gdb for backtrace]
    F --> G[Find root cause]
```

1. Read the service journal with `journalctl -u <service> -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime.
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command:
   ```
   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
   ```
   `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.

### Operational quirks

- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw vLLM ~5-10 minutes (38 GB checkpoint). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).

### Dankclaw memory model

The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported
by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable
SVM pool). The GPU node reports `local_mem_size 0`, so all allocations are
GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
`dmidecode` for the GPU budget; they show the CPU-side carve-out. This is
intentional. Do not change it.

## Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

> User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart

## Summary

| Context | Action |
|---------|--------|
| Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. |
| Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |