---
name: container-config-drift-crashloop
description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
---

# A container that restarts forever while its definition is correct

## When to use

A container is crash-looping or perpetually "unhealthy", but the file that
defines it (compose service, unit, deploy script) looks right. Also when a
service has been quietly down for days and nobody noticed, because a restart
policy kept something running.

The cause is usually **config drift**: the running container was created from an
older definition, and the restart policy restarts it instead of recreating it.
A compose/unit edit does not touch an existing container.

## Diagnose (5 minutes, in this order)

1. **Who started it?** The label names the owner, not your guesswork:

   ```
   podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
   ```

   `podman-compose@…` means compose owns it, whatever unit files exist.
   Compare `created` with when the definition last changed. A container older
   than the edit that should have replaced it is the bug.

2. **What is it ACTUALLY running?** Read the argv, not the summary fields:

   ```
   podman inspect <name> --format '{{.Config.CreateCommand}}'
   ```

   Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
   flags are present — a false negative that can send you hunting the wrong
   root cause. `CreateCommand` is the truth; grep it for the flag you care
   about (`--device`, `--group-add`, `-e`, mounts).

3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
   A capability error (`No HIP GPUs are available`) alongside a missing device
   flag in step 2 confirms drift rather than a code or image fault. Count the
   damage: `podman inspect --format '{{.RestartCount}}'`.

## Fix

Quickest unblock: recreate from the definition (`podman compose up -d
--force-recreate <svc>`), which is what should have happened automatically.

Durable fix, when the host is governed by a repo that requires boot-persistent
units (this one does): migrate the service to a Quadlet. Order that worked:

1. build the image and `install` the unit file to `~/.config/containers/systemd/`
2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
   leftover `~/.config/systemd/user/<svc>.service` collides by name with the
   unit the Quadlet generator produces
3. `podman rm -f <container>` — the stale container holds the name the unit's
   container needs, and it carries the old config
4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
5. delete the compose service so only one owner remains per container name

## Verify the capability, not the status

`Up` and even `healthy` can be true of a half-configured container. Check the
thing that was broken, from inside:

```
podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
ss -ltn | grep <port>            # confirm the bind addresses, not the intent
```

And confirm boot persistence the way the generator exposes it:
`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
explicit `enable` fails on generated units.

## Related reading in this repo

- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice