From 447286c2a1c66a0f691d77f7ab984d13febb3bc2 Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Thu, 13 Aug 2026 18:08:56 -0700
Subject: [PATCH] docs(AGENTS.md): add runtime & troubleshooting knowledge from
 the TEMP/HIP incident

Four sections: env var naming rule (HIP collision, TEMP/HOST/GPU_MAX_HEAP_SIZE),
crash-loop diagnostic procedure, operational quirks (sudo -S, pkill -x, slow
load, page-cache confound), and the Dankclaw memory model (96 GiB SVM budget).
---
 AGENTS.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/AGENTS.md b/AGENTS.md
index 955bdcb..3498668 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -45,6 +45,52 @@ Target map:
 | Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
 | Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 |
 
+## Runtime & Troubleshooting
+
+### Env var naming rule — HIP runtime collision
+
+The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with `set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every model load. Commit `227ce01` fixed it by renaming the sampling variable to `TEMPERATURE`.
+
+Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for collisions with this command:
+
+```
+strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
+```
+
+### Crash-loop diagnostic procedure
+
+When a llama service segfaults during model load, follow these steps.
+
+```mermaid
+graph TD
+    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
+    B --> C{Segfault in libamdhip64?}
+    C -- yes --> D[Bisect script vs raw binary]
+    C -- no --> E[Check config, memory, model files]
+    D --> F[Run gdb for backtrace]
+    F --> G[Find root cause]
+```
+
+1. Read the service journal with `journalctl -u <service> -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime.
+2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
+3. Run gdb for the backtrace with this command:
+   ```
+   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-llama-frank-large.sh
+   ```
+   `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
+4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
+
+### Operational quirks
+
+- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
+- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
+- Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process.
+- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing.
+
+### Dankclaw memory model
+
+The GPU budget on Dankclaw is the rocm-smi figure: 96 GiB addressable VRAM (SVM). The GPU node reports `local_mem_size 0`, so all allocations are GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or `dmidecode` for the GPU budget; they show the CPU-side carve-out. This is intentional. Do not change it.
+
 ## Why
 
 Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.