command-center/agent/managed-skills/container-config-drift-crashloop/SKILL.md

name, description

name	description
container-config-drift-crashloop	Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order.

A container that restarts forever while its definition is correct

When to use

A container is crash-looping or perpetually "unhealthy", but the file that defines it (compose service, unit, deploy script) looks right. Also when a service has been quietly down for days and nobody noticed, because a restart policy kept something running.

The cause is usually config drift: the running container was created from an older definition, and the restart policy restarts it instead of recreating it. A compose/unit edit does not touch an existing container.

Diagnose (5 minutes, in this order)

1. Who started it? The label names the owner, not your guesswork:

   podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
   

   podman-compose@… means compose owns it, whatever unit files exist. Compare created with when the definition last changed. A container older than the edit that should have replaced it is the bug.

2. What is it ACTUALLY running? Read the argv, not the summary fields:

   podman inspect <name> --format '{{.Config.CreateCommand}}'
   

   Trap: {{.HostConfig.Devices}} prints an empty list even when --device flags are present — a false negative that can send you hunting the wrong root cause. CreateCommand is the truth; grep it for the flag you care about (--device, --group-add, -e, mounts).

3. What does the process say? podman logs <name> | grep -iE 'error|cannot|Runtime'. A capability error (No HIP GPUs are available) alongside a missing device flag in step 2 confirms drift rather than a code or image fault. Count the damage: podman inspect --format '{{.RestartCount}}'.

Fix

Quickest unblock: recreate from the definition (podman compose up -d --force-recreate <svc>), which is what should have happened automatically.

Durable fix, when the host is governed by a repo that requires boot-persistent units (this one does): migrate the service to a Quadlet. Order that worked:

1. build the image and install the unit file to ~/.config/containers/systemd/
2. stop/disable the legacy unit and move its file to ~/.AGENTTRASH/ — a leftover ~/.config/systemd/user/<svc>.service collides by name with the unit the Quadlet generator produces
3. podman rm -f <container> — the stale container holds the name the unit's container needs, and it carries the old config
4. systemctl --user daemon-reload && systemctl --user restart <svc>.service
5. delete the compose service so only one owner remains per container name

Verify the capability, not the status

Up and even healthy can be true of a half-configured container. Check the thing that was broken, from inside:

podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
ss -ltn | grep <port>            # confirm the bind addresses, not the intent

And confirm boot persistence the way the generator exposes it: systemctl --user is-enabled <svc>.service → generated is healthy; an explicit enable fails on generated units.

Related reading in this repo

• AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
• serverAppConfigs-docs/Dankclaw/ComfyUI.md — the 9,146-restart incident
• serverAppConfigs-docs/Infrastructure/Observability.md — same class, twice