frank-primary: pin PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
The deployed unit failed its first boot with the engine core OOMing during profile/graph capture: "Tried to allocate 662.00 MiB. GPU 0 has a total capacity of 31.86 GiB of which 240.00 MiB is free ... try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True". The expert-offload plan fills VRAM to its modelled headroom by design, so the default allocator's fragmentation leaves the capture phase nothing — every successful bring-up probe (both scratch-port and 7770) carried this env, and the repo recipe dropped it. Also pins the seam: the setting is part of the qualified shape, not an optional tuning.
This commit is contained in:
1 parent
f2bcf84e88
commit
a3e1c97408
2 files changed
+10
No files matched your search
@@ -104,6 +104,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|||||||
--cap-add SYS_PTRACE \
|
--cap-add SYS_PTRACE \
|
||||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||||
"$FRANK_PRIMARY_IMAGE" /models \
|
"$FRANK_PRIMARY_IMAGE" /models \
|
||||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||||
|
|||||||
@@ -106,6 +106,15 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
|||||||
grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
|
grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "expandable segments is ON — the cache arena leaves no slack" {
|
||||||
|
# The expert-offload plan fills VRAM to the modelled headroom, so the
|
||||||
|
# profile/graph-capture phase has ~0.2 GiB to work with and the default
|
||||||
|
# allocator's fragmentation fails it: measured 2026-09-30 on the deployed
|
||||||
|
# unit, "Tried to allocate 662.00 MiB ... 240.00 MiB is free" from the
|
||||||
|
# engine core while the identical shape booted with this env set.
|
||||||
|
grep -q 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' "$SERVE"
|
||||||
|
}
|
||||||
|
|
||||||
@test "expert offload is ON — the payload cannot fit VRAM without it" {
|
@test "expert offload is ON — the payload cannot fit VRAM without it" {
|
||||||
# 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
|
# 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
|
||||||
# the engine has no home for the expert tensors at all; the arena
|
# the engine has no home for the expert tensors at all; the arena
|
||||||
|
|||||||
Reference in new issue
Block a user