From 0afdd9d7b58877921a9ba2ee2f558a7a1ddf5ce3 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 13 Aug 2026 18:04:24 -0700 Subject: [PATCH] docs: document TEMP env var / HIP runtime crash incident in troubleshooting --- docs/deployment.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deployment.md b/docs/deployment.md index c5bc0bb..aee9795 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -129,6 +129,16 @@ If the health check fails after a deploy: 3. Confirm that the GPU has free memory. 4. Check the service status with `systemctl status `. +## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13) + +On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring. + +Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not. + +Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged. + +If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. + ## See Also - [AGENTS.md](../AGENTS.md) — the git workflow rules