---
name: fleet-container-ownership-audit
description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them."
---

# Finding the containers that will not survive a reboot

Use when auditing a Podman fleet for boot persistence, when a service is
"up" but stopped working at some past reboot, or when a compose-managed stack is
being converted to Quadlet units. This is the *detection* half; the conversion
checklist is separate.

## The one-line test

Every running container on a boot-persistent host should carry a
`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and
rootless Podman does not restart bare containers after the explicit stop that
shutdown performs — so it will be missing tomorrow morning.

```bash
for c in $(podman ps --format '{{.Names}}'); do
  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done
```

Compare the result against the host's installed units
(`ls ~/.config/containers/systemd/*.container`) and the repo's
`<host>/<service>/quadlet/`. Three outcomes, three meanings:

- **label present** → the unit owns it; boot persistence comes from `[Install]
  WantedBy=default.target` plus `loginctl show-user <user> -p Linger` = yes.
- **label absent, compose service exists** → the classic gap: it dies at reboot.
  Also check the publish address while you are there; compose forms often bind
  the wildcard, which puts an unauthenticated UI/API on every interface.
- **label present AND a compose service with the same name** → a second owner.
  The runtime is fine, but a stray `podman compose up` can create a duplicate
  that fights the unit for the container name and the port. Remove the compose
  service and leave an explanatory comment; that is the repo's convention.

## Traps that make this hard to see

- **Editing a compose service does not replace a running container.**
  `restart: unless-stopped` will happily keep an old container — old image, old
  device list, old mounts — alive forever. One instance ran 9,146 restart
  attempts on stale config before anyone looked. If the container predates the
  edit, recreate it explicitly; do not debug the service.
- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.**
  It can read empty on a container that demonstrably has the devices. Read the
  real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside:
  `podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available())'`.
- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape
  that flips from `connection refused` to `unsupported Content-Type "text/html"`
  means the service came up and only the scrape is wrong. Read the error, not
  just up/down — and never "fix" a red probe by deleting the job, because an
  absent scrape and a healthy one look identical.
- **A hand-written `~/.config/systemd/user/<svc>.service` that shells out to
  `podman compose up` collides by name with the Quadlet the generator produces
  for `<svc>.container`.** Retire it to a dated trash directory before
  installing the unit.
- **`is-enabled` saying `generated` is the healthy state** for Quadlet units;
  `systemctl --user enable` fails on them.

## Migration shape that works

1. Unit file in the repo at `<host>/<service>/quadlet/<service>.container`:
   `Image=` with the `localhost/` prefix for locally built images, `Pull=never`,
   explicit `PublishPort` (loopback unless the service is the documented LAN
   exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed,
   `Restart=always`, `[Install] WantedBy=default.target`.
2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator
   splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle
   that fails silently.
3. `deploy_<svc>()` in the remote handler: install the unit, retire any
   compose-era unit into `~/.AGENTTRASH/`, `podman rm -f <container>` so the
   unit's own can take the name, daemon-reload, restart, wait for health.
4. Delete the compose service (one owner per container name) and leave a comment
   naming the unit, so the next reader does not re-add it.
5. Planner: add a path rule (`<host>/<dir>/**` -> service) **and** a
   `service_is_allowed` entry, or the shared dispatcher rejects the service
   before anything runs.
6. Migrate the seams: tests that read the compose file for devices, command
   flags or groups must read the unit instead — and resolve file paths from the
   tooling's own declarations, never from `find ... | head -1`.

## When the last service leaves a compose file

The repo's compose-validity seam requires at least one service, so a file whose
services have all migrated fails its own test. Either keep the file as
comments and relax that seam deliberately, or delete the file and fix the
planner's compose-diff path — decide with the owner, because the deploy tooling
reads it.