command-center/agent/managed-skills/stale-compose-container-quadlet-migration/SKILL.md

name, description

name	description
stale-compose-container-quadlet-migration	Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit.

Diagnose a crash-looping container after a reboot: stale compose container

When to use

A service that worked before a reboot now crash-loops (or is Exited), its config file looks correct, and the failure message blames something the config does declare — e.g. RuntimeError: No HIP GPUs are available for a container whose compose service passes /dev/kfd.

Root-cause pattern

The running container was created by an EARLIER revision of the service (before the missing lines were added) and restart: unless-stopped keeps resurrecting it:

• podman compose up -d does not recreate an existing container because its declaration changed — it only reconciles what compose tracks.
• So the container keeps its original flags forever, and every restart mode (unless-stopped, always, podman-compose unit, quadlet) faithfully keeps restarting a container that can never work.
• Symptom shape: high RestartCount, exit code 0 or 1, a config that looks right, and a failure that contradicts the config.

Confirm in three commands

# 1. How long has it been looping, and what actually started it?
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'

# 2. The flags it REALLY has — this is ground truth. Do NOT trust
#    {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
#    containers that were started with --device and --group-add.
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'

# 3. Behavioural truth inside it
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'

Compare the container's created timestamp with when the config gained the missing lines (git log -p --follow <config>). Created-before-the-edit is the signature.

Fix

Recreating the compose container works, but on a repo-governed fleet the fix is to give the service a repo-tracked, boot-persistent Quadlet unit (every container should already be one), then retire the compose path:

1. docker/quadlet/<name>.container with the full run posture: image (use Image=localhost/<tag> for locally built images + Pull=never), AddDevice=, GroupAdd=keep-groups, Volume=, PublishPort=127.0.0.1:... for an unguarded service, HealthCmd=, Restart=always, and [Install] WantedBy=default.target.
2. Healthcheck as a mounted file, not inline: the podman 5.x v1 generator splits HealthCmd on spaces into an argv array, so python3 -c "..." becomes a silent quoting puzzle.
3. Delete the compose service (one owner per container name) and make the deployer build → install_quadlet → retire any legacy hand-written unit into ~/.AGENTTRASH/ → podman rm -f <name> (so the unit's container can take the name) → restart → wait_for_health.
4. Wire the planner (<target>/<svc>/** → service) and the dispatcher allowlist (<target>:<svc>), or the deploy refuses the service.
5. Update the seams to read the unit rather than compose for devices, command and groups — and resolve build inputs from the deployer's declaration, never from a find | head -1 directory listing.

Verify behaviourally, not structurally

systemctl --user is-enabled → generated, container healthy, RestartCount stops climbing, the port answers (curl -o /dev/null -w '%{http_code}'), and the in-container probe reports the device (hip available: True | Radeon 8060S Graphics).