docs: document TEMP env var / HIP runtime crash incident in troubleshooting

2026-08-13 18:04:24 -07:00
parent 227ce01417
commit 0afdd9d7b5
1 changed files with 10 additions and 0 deletions
@@ -129,6 +129,16 @@ If the health check fails after a deploy:
 3. Confirm that the GPU has free memory.
 4. Check the service status with `systemctl status <service>`.

+## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
+
+On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring.
+
+Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.
+
+Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged.
+
+If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
+
 ## See Also

 - [AGENTS.md](../AGENTS.md) — the git workflow rules