frank-primary vllm: orphan-container self-heal + HOME + stop/rm order
Unit restart SIGKILLs conmon before --rm cleanup runs, leaving the container name registered: every restart then dies on name-in-use while the orphan holds the port and GPUs (observed live 2026-09-27, NRestarts=6 with the container Up and the unit auto-restarting). The serve script now clears its own name before exec podman run; the deployer stops the unit and removes the container BEFORE waiting on the port. HOME inside the container pointed at the root-owned WORKDIR /app, so vLLM's registry cache write hit PermissionError; HOME now points at the cache volume. The vLLM cutover itself is blocked on a structural wall (AMD Qwen4Exp impl creates the 51.2B PLE table as a bf16 on-GPU param: 47.69 GiB transient on a 31.86 GiB card; the pinned-host UVA path that fits this hardware is CUDA-gated). llama.cpp is restored live at :7770; host HEAD unchanged.
This commit is contained in:
@@ -1223,11 +1223,11 @@ PY
|
||||
# A stale container with the same name makes `podman run --name` fail at
|
||||
# start; remove any leftover from a failed previous attempt.
|
||||
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
|
||||
# On a re-deploy the predecessor is THIS unit (the llama.cpp unit until
|
||||
# the 2026-09-27 cutover, the vLLM unit after it): stop it before
|
||||
# waiting on the port, or the wait deadlocks on the server it just
|
||||
# stopped serving.
|
||||
# A stopped unit can leave its container running: systemd's cgroup kill
|
||||
# races conmon's --rm cleanup, and the orphan holds the port and GPUs.
|
||||
# Stop the unit, then remove the container BEFORE waiting on the port.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
podman rm -f "$LLM_PRIMARY_CONTAINER" >/dev/null 2>&1 || true
|
||||
# Nothing starts until the port is released: the lucebox cutover
|
||||
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
|
||||
Reference in New Issue
Block a user