serverAppConfigs/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Quadlet Lifecycle	The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.	podman quadlet systemd boot-persistence compose drift	2026-10-05T00:00:00-07:00

Quadlet Lifecycle

Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed, working container MUST have a self-healing, boot-persistent Quadlet unit — .container file in the repo (<host>/<svc>/quadlet/), installed by the deployer, [Install] WantedBy=default.target, Restart=always, linger on. A container that runs only from compose dies silently at reboot. This page is the operating manual for that lifecycle.

1. Ownership audit — find the containers that will not survive a reboot

for c in $(podman ps --format '{{.Names}}'); do
  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done

• label present → the unit owns it; persistence = [Install] + loginctl show-user <user> -p Linger = yes.
• label absent, compose service exists → the classic gap: dies at reboot. Check the publish address while there — compose forms often bind the wildcard (an unauthenticated UI/API on every interface).
• label present AND a compose service with the same name → two owners; a stray podman compose up can create a duplicate that fights the unit for the name and the port. Remove the compose service, leave an explanatory comment (repo convention).

2. Cutover: compose service → Quadlet unit

Checklist (precedents to copy: dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/, dankclaw/observability/quadlet/):

1. Unit file at <host>/<svc>/quadlet/<name>.container, mirroring compose line for line: Image= (localhost/ prefix for locally built images + Pull=never), ContainerName= (Caddy and scrape configs address containers by name), Network=, User=0:0 when the image's own user cannot write a rootless-owned bind mount, Volume= (:ro preserved), Exec= for compose command:, PublishPort=, health keys, [Service] Restart=always + RestartSec=10, [Install] WantedBy=default.target. Never systemctl --user enable the generated unit — [Install] + linger IS the persistence (is-enabled = generated is healthy; explicit enable fails with "transient or generated").
2. Exposure: publish on 127.0.0.1 unless something off-host genuinely needs it (Caddy reaches containers by name over the shared network; the wildcard publish is a guard-chain bypass).
3. Handler in scripts/deploy-remote.sh: install_quadlet <dir> <name> → systemctl --user restart <name>.service → wait_for_health; retire any legacy hand-written unit into ~/.AGENTTRASH/ (a leftover ~/.config/systemd/user/<svc>.service that shells out to podman compose up collides BY NAME with the generated unit); podman rm -f <container> so the unit's container can take the name.
4. Planner: map <host>/<svc>/** → service in scripts/deploy-plan.py AND add a service_is_allowed entry; keep SERVICE_ORDER dependencies ordered. Add a test-changed case above the generic */quadlet/*.container fallback.
5. Retire the compose entry in the SAME change — two owners is how a stale duplicate comes back.
6. Seam test tests/<svc>_quadlet.bats: boot persistence, restart policy, no wildcard publish, container name matches Caddy/scrape references, health command, deployer installs+restarts+waits, never rm, never enable. Migrate any older assertion that read the compose file for devices/flags.
7. Deploy --service <name>; prove: is-active/is-enabled = active/generated, container healthy, endpoint answers, list-dependencies default.target shows the unit, off-host probes on previously published ports now refuse, and the container no longer carries io.podman.compose.project.

Traps that cost real time:

• HealthCmd = CMD-SHELL (podman runs /bin/sh -c); a distroless image has no shell → the check fails forever. --health-cmd won't take exec form either. Put HEALTHCHECK … CMD ["binary","arg"] in the image and build with --format docker — the default OCI format has no HEALTHCHECK field and drops it with only a warning.
• Never point a unit's EnvironmentFile at the repo-root .env — it is a gluetun env carrying HOME=/root; systemd sets HOME for the service and the unit dies with stat /root/.config: permission denied. Use a per-service env under /home/dank/Containers/<svc>/env, seeded by the handler with a single-variable filter, validated so a missing secret fails loudly.
• Network=host exporters scraped by containerized Prometheus must bind the host's LAN address (containers cannot reach host loopback) PLUS 127.0.0.1 for the homepage probe — two --web.listen-address flags, never the wildcard.
• alloy 1.x refuses the ui config block; the unit must pass --server.http.listen-addr=0.0.0.0:12345 before its port can be published.
• GPU services: AddDevice= + GroupAdd=keep-groups (user-unit spawns carry no supplementary groups).
• When the last service leaves a compose file, the compose-validity seam fails (requires ≥1 service): keep the file as comments and relax the seam deliberately, or delete the file and fix the planner's compose-diff path — decide with the owner, the deploy tooling reads it.

3. Drift / crash-loop diagnosis (5 minutes, in order)

Symptom: crash-looping or perpetually unhealthy container whose DEFINITION looks right; or a service quietly down for days because a restart policy kept something running. Cause: the running container was created from an OLDER definition — a compose/unit edit does not touch an existing container, and the restart policy faithfully resurrects it forever.

1. Who started it: podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'; compare created with when the definition changed (git log -p --follow <config>). Created-before-the-edit is the signature.
2. What it is ACTUALLY running: {{.Config.CreateCommand}} is the truth. {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}} print empty/0 even for containers started with --device/--group-add — a false negative that sends you hunting the wrong root cause.
3. What the process says: podman logs <name> | grep -iE 'error|cannot|Runtime' (a capability error like No HIP GPUs are available beside a missing device flag confirms drift). Count the damage: {{.RestartCount}} — one instance reached 9,146 restart attempts on stale config.

Quickest unblock: podman compose up -d --force-recreate <svc>. Durable fix = the §2 cutover. Verify the capability, not the status: podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"; ss -ltn | grep <port> for the actual bind addresses. ComfyUI incident: 2026-09-16, stale device-less container, 9,146 restarts. History repeats: 2026-08-26 Caddy (no unit), 2026-09-11 five observability services.

4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)

• Hard-fail keys — the whole unit silently fails to generate (systemctl --user cat <unit> shows "could not be found"): any dash-prefixed [Container] option (--shm-size=… → unsupported key … in group 'Container'); Command= is unsupported, use Exec=.
• HealthCmd space-splitting: the v1 generator splits HealthCmd on spaces into the healthcheck argv; a quoted probe /bin/sh -c "pg_isready …" lands as argv token "pg_isready → command-not-found → container reads unhealthy while the DB is up. HealthCmd must contain only space-free tokens (HealthCmd=pg_isready -d authentik -U authentik). If health reads unhealthy but logs look fine, compare the baked {{.Config.Healthcheck.Test}} array against the intended command before believing the status.
• Probe a new key before shipping: drop a throwaway probe.container, daemon-reload, journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key", systemctl --user cat probe.service (shows the generated ExecStart verbatim), then remove + reload.

5. Rootless Quadlet log retrieval

Step 0 — establish ownership: systemctl --user status <name>.service (works ⇒ rootless user unit); loginctl show-user <user> -p Linger. If --user knows the unit, EVERY command runs as that user with NO sudo.

Need	Command
Current instance tail/follow	podman logs --tail 200 / -f <name>
Across restarts/recreates	journalctl --user -u <name>.service -n 200 --no-pager
Time-bounded / crash-only	--since "14:30" / -p warning
Kernel-side evidence (OOM, SVM)	sudo journalctl -k --since …

Quadlet units commonly run --replace --rm: podman logs is destroyed on recreation — the journal is the only durable record. Traps: sudo podman logs → "no such container" (root's store does not contain rootless containers; same for sudo podman ps); a system-level journalctl -u <name> returns nothing for a user unit; during first-start weight downloads health reports unhealthy BY DESIGN (check the payload file is growing). Verbose vendor switches are a config change — through git and deploy.sh, never a live env poke.

6. Compose bring-up with ground truth (rootless hosts)

1. Audit reality before pressing buttons: podman ps -a runs per-user — check every namespace (sudo podman ps -a) and native units (systemctl list-units) before declaring a service absent. Compare compose mount sources against the real filesystem — compose files name phantom paths (e.g. /mnt/media/* vs the real LUKS array at /mnt/entropy). Fix compose through git first, deploy second.
2. Pre-create bind-mount sources (sudo mkdir -p + chown 1000:1000) or creation fails statfs …: no such file or directory.
3. Staged bring-up: infra first (up -d mariadb redis), poll {{.State.Health.Status}}, then the rest. Fresh containers answer 000/refused for the first seconds — poll.
4. Never guess image tags: ghcr.io answers 403 on the bearer-token request for nonexistent/private repos ("manifest unknown"/token len 0 = repo or tag does not exist, not auth). Query the registry API (ghcr.io/token?scope=repository:<ns>/<repo>:pull → /v2/<ns>/<repo>/tags/list).
5. Read the interface from inside the image before guessing flags/envs: podman exec <ctr> cat /z/cpp.sh etc. HTTP 500 with a styled error page means RUNNING-but-misconfigured, not down. Copyparty ac ground truth: entrypoint /bin/ash /z/cpp.sh -c /z/initcfg; accounts/volumes are ARGV flags (-a USER:PASS, -v SRC:DST:PERMS), not envs — and with none configured it serves cwd read/write for everyone (an unauthenticated world-writable share).
6. Remote-shell quoting: zsh eats bare ===/? tokens; a zero-match from a mangled pattern is not absence — re-verify the target exists.
7. user: "1000:1000" SIGSEGV at boot = unmapped uid in the rootless userns (host 1000 == container 0): drop the user line, run as container-root.

Related

• Deployment Flow — the deployer that installs these units
• Stale Signal Audit — the audit that catches drift's victims
• Container Network Namespaces — reachability between containers
• Dankclaw/ComfyUI — the 9,146-restart incident
• Infrastructure/Observability — same class, twice