serverAppConfigs/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

---
type: reference
title: Quadlet Lifecycle
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
timestamp: 2026-10-05T00:00:00-07:00
---

# Quadlet Lifecycle

Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
working container MUST have a self-healing, boot-persistent Quadlet unit —
`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
A container that runs only from compose dies silently at reboot. This page
is the operating manual for that lifecycle.

## 1. Ownership audit — find the containers that will not survive a reboot

```bash
for c in $(podman ps --format '{{.Names}}'); do
  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done
```

- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
  show-user <user> -p Linger` = yes.
- **label absent, compose service exists** → the classic gap: dies at reboot.
  Check the publish address while there — compose forms often bind the
  wildcard (an unauthenticated UI/API on every interface).
- **label present AND a compose service with the same name** → two owners; a
  stray `podman compose up` can create a duplicate that fights the unit for
  the name and the port. Remove the compose service, leave an explanatory
  comment (repo convention).

## 2. Cutover: compose service → Quadlet unit

Checklist (precedents to copy:
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
`dankclaw/observability/quadlet/`):

1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
   line for line: `Image=` (`localhost/` prefix for locally built images +
   `Pull=never`), `ContainerName=` (Caddy and scrape configs address
   containers by name), `Network=`, `User=0:0` when the image's own user
   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
   `Exec=` for compose `command:`, `PublishPort=`, health keys,
   `[Service] Restart=always` + `RestartSec=10`, `[Install]
   WantedBy=default.target`. **Never `systemctl --user enable` the generated
   unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
   is healthy; explicit enable fails with "transient or generated").
2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
   it (Caddy reaches containers by name over the shared network; the wildcard
   publish is a guard-chain bypass).
3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
   `systemctl --user restart <name>.service` → `wait_for_health`; retire any
   legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
   `~/.config/systemd/user/<svc>.service` that shells out to
   `podman compose up` collides BY NAME with the generated unit);
   `podman rm -f <container>` so the unit's container can take the name.
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
   AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
   ordered. Add a `test-changed` case above the generic
   `*/quadlet/*.container` fallback.
5. Retire the compose entry in the SAME change — two owners is how a stale
   duplicate comes back.
6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
   wildcard publish, container name matches Caddy/scrape references, health
   command, deployer installs+restarts+waits, never `rm`, never `enable`.
   Migrate any older assertion that read the compose file for devices/flags.
7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
   active/generated, container `healthy`, endpoint answers,
   `list-dependencies default.target` shows the unit, off-host probes on
   previously published ports now refuse, and the container no longer carries
   `io.podman.compose.project`.

**Traps that cost real time:**

- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
  no shell → the check fails forever. `--health-cmd` won't take exec form
  either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
  with `--format docker` — the default OCI format has no HEALTHCHECK field
  and drops it with only a warning.
- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
  gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
  the unit dies with `stat /root/.config: permission denied`. Use a
  per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
  handler with a single-variable filter, validated so a missing secret fails
  loudly.
- `Network=host` exporters scraped by containerized Prometheus must bind the
  host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
  for the homepage probe — two `--web.listen-address` flags, never the
  wildcard.
- alloy 1.x refuses the `ui` config block; the unit must pass
  `--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
  no supplementary groups).
- When the last service leaves a compose file, the compose-validity seam
  fails (requires ≥1 service): keep the file as comments and relax the seam
  deliberately, or delete the file and fix the planner's compose-diff path —
  decide with the owner, the deploy tooling reads it.

## 3. Drift / crash-loop diagnosis (5 minutes, in order)

Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
looks right; or a service quietly down for days because a restart policy kept
something running. Cause: the running container was created from an OLDER
definition — a compose/unit edit does not touch an existing container, and
the restart policy faithfully resurrects it forever.

1. **Who started it:** `podman inspect <name> --format 'unit={{index
   .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
   created={{.Created}}'`; compare `created` with when the definition changed
   (`git log -p --follow <config>`). Created-before-the-edit is the signature.
2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
   `{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
   for containers started with `--device`/`--group-add` — a false negative
   that sends you hunting the wrong root cause.
3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
   (a capability error like `No HIP GPUs are available` beside a missing
   device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
   instance reached 9,146 restart attempts on stale config.

Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
= the §2 cutover. **Verify the capability, not the status:** `podman exec
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.

## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)

- **Hard-fail keys** — the whole unit silently fails to generate
  (`systemctl --user cat <unit>` shows "could not be found"): any
  dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
  in group 'Container'`); `Command=` is unsupported, use `Exec=`.
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
  spaces into the healthcheck argv; a quoted probe
  `/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
  command-not-found → container reads **unhealthy while the DB is up**.
  HealthCmd must contain only space-free tokens
  (`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
  unhealthy but logs look fine, compare the baked
  `{{.Config.Healthcheck.Test}}` array against the intended command before
  believing the status.
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
  `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
  "unsupported key"`, `systemctl --user cat probe.service` (shows the
  generated ExecStart verbatim), then remove + reload.

## 5. Rootless Quadlet log retrieval

Step 0 — establish ownership: `systemctl --user status <name>.service`
(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
`--user` knows the unit, EVERY command runs as that user with NO sudo.

| Need | Command |
|---|---|
| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |

Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
recreation** — the journal is the only durable record. Traps:
`sudo podman logs` → "no such container" (root's store does not contain
rootless containers; same for `sudo podman ps`); a system-level
`journalctl -u <name>` returns nothing for a user unit; during first-start
weight downloads health reports unhealthy BY DESIGN (check the payload file
is growing). Verbose vendor switches are a config change — through git and
deploy.sh, never a live env poke.

## 6. Compose bring-up with ground truth (rootless hosts)

1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
   check every namespace (`sudo podman ps -a`) and native units
   (`systemctl list-units`) before declaring a service absent. Compare
   compose mount sources against the real filesystem — compose files name
   phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
   `/mnt/entropy`). Fix compose through git first, deploy second.
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
   or creation fails `statfs …: no such file or directory`.
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
   `{{.State.Health.Status}}`, then the rest. Fresh containers answer
   000/refused for the first seconds — poll.
4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
   request for nonexistent/private repos ("manifest unknown"/token len 0 =
   repo or tag does not exist, not auth). Query the registry API
   (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
   `/v2/<ns>/<repo>/tags/list`).
5. **Read the interface from inside the image** before guessing flags/envs:
   `podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
   means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
   entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
   flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
   configured it serves cwd read/write for everyone (an unauthenticated
   world-writable share).
6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
   mangled pattern is not absence — re-verify the target exists.
7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
   (host 1000 == container 0): drop the user line, run as container-root.

## Related

- [[Deployment Flow]] — the deployer that installs these units
- [[Stale Signal Audit]] — the audit that catches drift's victims
- [[Container Network Namespaces]] — reachability between containers
- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
- [[Infrastructure/Observability]] — same class, twice