serverAppConfigs/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Quadlet Lifecycle	The Quadlet discipline: ownership audits, compose cutover, drift and crash-loop triage, generator quirks, log retrieval.	podman quadlet systemd boot-persistence compose drift	2026-10-05T00:00:00-07:00

Quadlet Lifecycle

Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed, working container MUST have a self-healing, boot-persistent Quadlet unit — .container file in the repo (<host>/<svc>/quadlet/), installed by the deployer, [Install] WantedBy=default.target, Restart=always, linger on. A container that runs only from compose dies silently at reboot. This page is the operating manual for that lifecycle.

1. Ownership audit — find the containers that will not survive a reboot

for c in $(podman ps --format '{{.Names}}'); do
  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done

• label present → the unit owns it; persistence = [Install] + loginctl show-user <user> -p Linger = yes.
• label absent, compose service exists → the classic gap: dies at reboot. Check the publish address while there — compose forms often bind the wildcard (an unauthenticated UI/API on every interface).
• label present AND a compose service with the same name → two owners; a stray podman compose up can create a duplicate that fights the unit for the name and the port. Remove the compose service, leave an explanatory comment (repo convention).

2. Cutover: compose service → Quadlet unit

Checklist (precedents to copy: dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/ dankclaw/observability/quadlet/):

1. Unit file at <host>/<svc>/quadlet/<name>.container, mirroring compose line for line: Image= (localhost/ prefix for locally built images + Pull=never), ContainerName= (Caddy and scrape configs address containers by name), Network=, User=0:0 when the image's own user cannot write a rootless-owned bind mount, Volume= (:ro preserved) Exec= for compose command:, PublishPort=, health keys, [Service] Restart=always + RestartSec=10, [Install] WantedBy=default.target. Never systemctl --user enable the generated unit — [Install] + linger IS the persistence (is-enabled = generated is healthy; explicit enable fails with "transient or generated").
2. Exposure: publish on 127.0.0.1 unless something off-host genuinely needs it (Caddy reaches containers by name over the shared network; the wildcard publish is a guard-chain bypass).
3. Handler in scripts/deploy-remote.sh: install_quadlet <dir> <name> → systemctl --user restart <name>.service → wait_for_health; retire any legacy hand-written unit into ~/.AGENTTRASH/ (a leftover ~/.config/systemd/user/<svc>.service that shells out to podman compose up collides BY NAME with the generated unit) podman rm -f <container> so the unit's container can take the name.
4. Planner: map <host>/<svc>/** → service in scripts/deploy-plan.py AND add a service_is_allowed entry; keep SERVICE_ORDER dependencies ordered. Add a test-changed case above the generic */quadlet/*.container fallback.
5. Retire the compose entry in the SAME change — two owners is how a stale duplicate comes back.
6. Seam test tests/<svc>_quadlet.bats: boot persistence, restart policy, no wildcard publish, container name matches Caddy/scrape references, health command, deployer installs+restarts+waits, never rm, never enable. Migrate any older assertion that read the compose file for devices/flags.
7. Deploy --service <name>; prove: is-active/is-enabled = active/generated, container healthy, endpoint answers, list-dependencies default.target shows the unit, off-host probes on previously published ports now refuse, and the container no longer carries io.podman.compose.project.

Traps that cost real time:

• HealthCmd = CMD-SHELL (podman runs /bin/sh -c); a distroless image has no shell → the check fails forever. --health-cmd won't take exec form either. Put HEALTHCHECK … CMD ["binary","arg"] in the image and build with --format docker — the default OCI format has no HEALTHCHECK field and drops it with only a warning.
• Never point a unit's EnvironmentFile at the repo-root .env — it is a gluetun env carrying HOME=/root; systemd sets HOME for the service and the unit dies with stat /root/.config: permission denied. Use a per-service env under /home/dank/Containers/<svc>/env, seeded by the handler with a single-variable filter, validated so a missing secret fails loudly.
• Network=host exporters scraped by containerized Prometheus must bind the host's LAN address (containers cannot reach host loopback) PLUS 127.0.0.1 for the homepage probe — two --web.listen-address flags, never the wildcard.
• alloy 1.x refuses the ui config block; the unit must pass --server.http.listen-addr=0.0.0.0:12345 before its port can be published.
• GPU services: AddDevice= + GroupAdd=keep-groups (user-unit spawns carry no supplementary groups).
• When the last service leaves a compose file, the compose-validity seam fails (requires ≥1 service): keep the file as comments and relax the seam deliberately, or delete the file and fix the planner's compose-diff path — decide with the owner, the deploy tooling reads it.

3. Drift / crash-loop diagnosis (5 minutes, in order)

Symptom: crash-looping or perpetually unhealthy container whose DEFINITION looks right; or a service quietly down for days because a restart policy kept something running. Cause: the running container was created from an OLDER definition — a compose/unit edit does not touch an existing container, and the restart policy faithfully resurrects it forever.

1. Who started it: podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'; compare created with when the definition changed (git log -p --follow <config>). Created-before-the-edit is the signature.
2. What it is ACTUALLY running: {{.Config.CreateCommand}} is the truth. {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}} print empty/0 even for containers started with --device/--group-add — a false negative that sends you hunting the wrong root cause.
3. What the process says: podman logs <name> | grep -iE 'error|cannot|Runtime' (a capability error like No HIP GPUs are available beside a missing device flag confirms drift). Count the damage: {{.RestartCount}} — one instance reached 9,146 restart attempts on stale config.

Quickest unblock: podman compose up -d --force-recreate <svc>. Durable fix = the §2 cutover. Verify the capability, not the status: podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())" ss -ltn | grep <port> for the actual bind addresses. ComfyUI incident: 2026-09-16, stale device-less container, 9,146 restarts. History repeats: 2026-08-26 Caddy (no unit), 2026-09-11 five observability services.

4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)

• Hard-fail keys abort the WHOLE BATCH, not one unit (podman 5.7.0, live 2026-10-21): one unsupported key — e.g. HostAddr= — makes the generator log unsupported key … in group 'Container' plus processing encountered some errors and generate NOTHING for that manager. The running containers keep going as orphans, so the failure first appears on the next deploy as Unit <x>.service not found for a service that is visibly up. Other offenders: any dash-prefixed [Container] option (--shm-size=…), Command= (use Exec=). The seam tests/quadlet_image_pins.bats fails the build on any generator-rejected key in any host's units. Also: an image whose ENTRYPOINT is the binary takes FLAGS ONLY in Exec= — repeating the binary makes it a positional arg (unexpected /bin/blackbox_exporter, live 2026-10-21).
• HealthCmd space-splitting: the v1 generator splits HealthCmd on spaces into the healthcheck argv; a quoted probe /bin/sh -c "pg_isready …" lands as argv token "pg_isready → command-not-found → container reads unhealthy while the DB is up. HealthCmd must contain only space-free tokens (HealthCmd=pg_isready -d authentik -U authentik). If health reads unhealthy but logs look fine, compare the baked {{.Config.Healthcheck.Test}} array against the intended command before believing the status.
• Probe a new key before shipping: drop a throwaway probe.container daemon-reload, journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key", systemctl --user cat probe.service (shows the generated ExecStart verbatim), then remove + reload.

4a. Image pins (fleet law, 2026-10-21)

Every registry image in a quadlet is digest-pinned (tag@sha256:…); locally built images (localhost/*, franksdank/*) are exempt — their build is the pin. A rolling tag lets a rebuild move under a running service with nobody approving it, and it makes a CVE audit ungradeable. Gate: tests/quadlet_image_pins.bats. Where upstream publishes no version tags (copyparty-ac stops at 1.19.15), pin :latest@sha256: and name the version the config label carries in the comment beside it.

4b. Resource control needs host permission (measured 2026-10-31, Dankserver)

IOWeight= in a rootless quadlet does nothing on its own, and nothing says so. The system manager delegates cpu memory pids to user@.service and not io; systemd drops a resource line it cannot honour without a word — the generator still emits IOWeight=10, the container still starts and answers its health check, and the cgroup simply has no io.weight file. Same family as RequiresMountsFor= (inert in the user manager) and ConditionPathIsMountPoint= (fail-dead): the line is accepted, the unit is happy, the setting is dead.

Two system drop-ins make it real, both placed by ensure_user_io_delegate in scripts/deploy-remote.sh (repo copies under dankserver/host/systemd/):

File	Effect
user.slice.d/io-accounting.conf → [Slice] IOAccounting=yes	puts io into user.slice/cgroup.subtree_control; lands immediately, no restart
user@1000.service.d/io-delegate.conf → [Service] DelegateControllers=cpu memory pids io	the user manager can only request controllers it was delegated; read when it starts, so it lands on the next reboot (restarting it kills every rootless container on the host)

Read the chain in this order — each line is the failure mode of the one after it:

cat /sys/fs/cgroup/user.slice/cgroup.subtree_control                    # wants: io
systemctl show user@1000.service -p DelegateControllers                 # wants: io
cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/\<unit\>.service/libpod-payload-*/io.weight

io missing from the first → the accounting drop-in is absent. Missing from the second → the host has not rebooted since the delegation landed (the deployer prints NOTE: io is not delegated … for exactly this). No file at the third → that unit carries no weight.

Two limits that survive the fix: a container's weight covers everything in its cgroup, so a library scan and a playback stream share one weight; and the comparison is made inside user.slice, so a rootless unit's weight cannot push it below a root unit (frank-archon). Shaping that pairing is the weight of user.slice itself — a fleet decision, not a unit line.

5. Rootless Quadlet log retrieval

Step 0 — establish ownership: systemctl --user status <name>.service (works ⇒ rootless user unit); loginctl show-user <user> -p Linger. If --user knows the unit, EVERY command runs as that user with NO sudo.

Need	Command
Current instance tail/follow	podman logs --tail 200 / -f <name>
Across restarts/recreates	journalctl --user -u <name>.service -n 200 --no-pager
Time-bounded / crash-only	--since "14:30" / -p warning
Kernel-side evidence (OOM, SVM)	sudo journalctl -k --since …

Quadlet units commonly run --replace --rm: podman logs is destroyed on recreation — the journal is the only durable record. Traps: sudo podman logs → "no such container" (root's store does not contain rootless containers; same for sudo podman ps); a system-level journalctl -u <name> returns nothing for a user unit; during first-start weight downloads health reports unhealthy BY DESIGN (check the payload file is growing). Verbose vendor switches are a config change — through git and deploy.sh, never a live env poke.

6. Compose bring-up with ground truth (rootless hosts)

1. Audit reality before pressing buttons: podman ps -a runs per-user — check every namespace (sudo podman ps -a) and native units (systemctl list-units) before declaring a service absent. Compare compose mount sources against the real filesystem — compose files name phantom paths (the legacy dankserver/compose.yaml still names /opt/jellyfin/* for a service whose roots moved to /home/dank/Containers/jellyfin). Fix compose through git first, deploy second.
2. Pre-create bind-mount sources (sudo mkdir -p + chown 1000:1000) or creation fails statfs …: no such file or directory.
3. Staged bring-up: infra first (up -d mariadb redis), poll {{.State.Health.Status}}, then the rest. Fresh containers answer 000/refused for the first seconds — poll.
4. Never guess image tags: ghcr.io answers 403 on the bearer-token request for nonexistent/private repos ("manifest unknown"/token len 0 = repo or tag does not exist, not auth). Query the registry API (ghcr.io/token?scope=repository:<ns>/<repo>:pull → /v2/<ns>/<repo>/tags/list).
5. Read the interface from inside the image before guessing flags/envs: podman exec <ctr> cat /z/cpp.sh etc. HTTP 500 with a styled error page means RUNNING-but-misconfigured, not down. Copyparty ac ground truth: entrypoint /bin/ash /z/cpp.sh -c /z/initcfg; accounts/volumes are ARGV flags (-a USER:PASS, -v SRC:DST:PERMS), not envs — and with none configured it serves cwd read/write for everyone (an unauthenticated world-writable share).
6. Remote-shell quoting: zsh eats bare ===/? tokens; a zero-match from a mangled pattern is not absence — re-verify the target exists.
7. user: "1000:1000" SIGSEGV at boot = unmapped uid in the rootless userns (host 1000 == container 0): drop the user line, run as container-root.

Related

• Deployment Flow — the deployer that installs these units
• Stale Signal Audit — the audit that catches drift's victims
• Container Network Namespaces — reachability between containers
• Dankclaw/ComfyUI — the 9,146-restart incident
• Infrastructure/Observability — same class, twice