frank-primary: pin PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

The deployed unit failed its first boot with the engine core OOMing during
profile/graph capture: "Tried to allocate 662.00 MiB. GPU 0 has a total
capacity of 31.86 GiB of which 240.00 MiB is free ... try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The expert-offload plan
fills VRAM to its modelled headroom by design, so the default allocator's
fragmentation leaves the capture phase nothing — every successful bring-up
probe (both scratch-port and 7770) carried this env, and the repo recipe
dropped it.

Also pins the seam: the setting is part of the qualified shape, not an
optional tuning.
This commit is contained in:
Frank
2026-09-30 05:13:50 -07:00
parent f2bcf84e88
commit a3e1c97408
2 changed files with 10 additions and 0 deletions
+1
View File
@@ -104,6 +104,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
"$FRANK_PRIMARY_IMAGE" /models \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \