frank-primary: pin PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

The deployed unit failed its first boot with the engine core OOMing during
profile/graph capture: "Tried to allocate 662.00 MiB. GPU 0 has a total
capacity of 31.86 GiB of which 240.00 MiB is free ... try setting
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The expert-offload plan
fills VRAM to its modelled headroom by design, so the default allocator's
fragmentation leaves the capture phase nothing — every successful bring-up
probe (both scratch-port and 7770) carried this env, and the repo recipe
dropped it.

Also pins the seam: the setting is part of the qualified shape, not an
optional tuning.
This commit is contained in:
Frank
2026-09-30 05:13:50 -07:00
parent f2bcf84e88
commit a3e1c97408
2 changed files with 10 additions and 0 deletions
+1
View File
@@ -104,6 +104,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
"$FRANK_PRIMARY_IMAGE" /models \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
+9
View File
@@ -106,6 +106,15 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
}
@test "expandable segments is ON — the cache arena leaves no slack" {
# The expert-offload plan fills VRAM to the modelled headroom, so the
# profile/graph-capture phase has ~0.2 GiB to work with and the default
# allocator's fragmentation fails it: measured 2026-09-30 on the deployed
# unit, "Tried to allocate 662.00 MiB ... 240.00 MiB is free" from the
# engine core while the identical shape booted with this env set.
grep -q 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' "$SERVE"
}
@test "expert offload is ON — the payload cannot fit VRAM without it" {
# 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
# the engine has no home for the expert tensors at all; the arena