---
name: stale-compose-container-quadlet-migration
description: "Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit."
---

# Diagnose a crash-looping container after a reboot: stale compose container

## When to use

A service that worked before a reboot now crash-loops (or is `Exited`), its
config file looks correct, and the failure message blames something the config
does declare — e.g. `RuntimeError: No HIP GPUs are available` for a container
whose compose service passes `/dev/kfd`.

## Root-cause pattern

The running container was created by an EARLIER revision of the service
(before the missing lines were added) and `restart: unless-stopped` keeps
resurrecting it:

- `podman compose up -d` does not recreate an existing container because its
  *declaration* changed — it only reconciles what compose tracks.
- So the container keeps its original flags forever, and every restart mode
  (`unless-stopped`, `always`, podman-compose unit, quadlet) faithfully keeps
  restarting a container that can never work.
- Symptom shape: high `RestartCount`, exit code 0 or 1, a config that looks
  right, and a failure that contradicts the config.

## Confirm in three commands

```bash
# 1. How long has it been looping, and what actually started it?
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'

# 2. The flags it REALLY has — this is ground truth. Do NOT trust
#    {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
#    containers that were started with --device and --group-add.
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'

# 3. Behavioural truth inside it
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'
```

Compare the container's `created` timestamp with when the config gained the
missing lines (`git log -p --follow <config>`). Created-before-the-edit is the
signature.

## Fix

Recreating the compose container works, but on a repo-governed fleet the fix is
to give the service a **repo-tracked, boot-persistent Quadlet unit** (every
container should already be one), then retire the compose path:

1. `docker/quadlet/<name>.container` with the full run posture: image (use
   `Image=localhost/<tag>` for locally built images + `Pull=never`),
   `AddDevice=`, `GroupAdd=keep-groups`, `Volume=`, `PublishPort=127.0.0.1:...`
   for an unguarded service, `HealthCmd=`, `Restart=always`, and
   `[Install] WantedBy=default.target`.
2. Healthcheck as a **mounted file**, not inline: the podman 5.x v1 generator
   splits `HealthCmd` on spaces into an argv array, so
   `python3 -c "..."` becomes a silent quoting puzzle.
3. Delete the compose service (one owner per container name) and make the
   deployer build → `install_quadlet` → retire any legacy hand-written unit
   into `~/.AGENTTRASH/` → `podman rm -f <name>` (so the unit's container can
   take the name) → restart → `wait_for_health`.
4. Wire the planner (`<target>/<svc>/**` → service) and the dispatcher
   allowlist (`<target>:<svc>`), or the deploy refuses the service.
5. Update the seams to read the unit rather than compose for devices, command
   and groups — and resolve build inputs from the deployer's declaration, never
   from a `find | head -1` directory listing.

## Verify behaviourally, not structurally

`systemctl --user is-enabled` → `generated`, container `healthy`,
`RestartCount` stops climbing, the port answers (`curl -o /dev/null -w
'%{http_code}'`), and the in-container probe reports the device
(`hip available: True | Radeon 8060S Graphics`).