Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
92 lines
5.0 KiB
Markdown
92 lines
5.0 KiB
Markdown
---
|
|
name: fleet-container-ownership-audit
|
|
description: "Audit a rootless-Podman fleet for containers that will not survive a reboot: owner-label detection, the stale-container and inspect-field traps, and the migration shape that fixes them."
|
|
---
|
|
|
|
# Finding the containers that will not survive a reboot
|
|
|
|
Use when auditing a Podman fleet for boot persistence, when a service is
|
|
"up" but stopped working at some past reboot, or when a compose-managed stack is
|
|
being converted to Quadlet units. This is the *detection* half; the conversion
|
|
checklist is separate.
|
|
|
|
## The one-line test
|
|
|
|
Every running container on a boot-persistent host should carry a
|
|
`PODMAN_SYSTEMD_UNIT` label. Anything without one is compose-only or bare, and
|
|
rootless Podman does not restart bare containers after the explicit stop that
|
|
shutdown performs — so it will be missing tomorrow morning.
|
|
|
|
```bash
|
|
for c in $(podman ps --format '{{.Names}}'); do
|
|
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
|
done
|
|
```
|
|
|
|
Compare the result against the host's installed units
|
|
(`ls ~/.config/containers/systemd/*.container`) and the repo's
|
|
`<host>/<service>/quadlet/`. Three outcomes, three meanings:
|
|
|
|
- **label present** → the unit owns it; boot persistence comes from `[Install]
|
|
WantedBy=default.target` plus `loginctl show-user <user> -p Linger` = yes.
|
|
- **label absent, compose service exists** → the classic gap: it dies at reboot.
|
|
Also check the publish address while you are there; compose forms often bind
|
|
the wildcard, which puts an unauthenticated UI/API on every interface.
|
|
- **label present AND a compose service with the same name** → a second owner.
|
|
The runtime is fine, but a stray `podman compose up` can create a duplicate
|
|
that fights the unit for the container name and the port. Remove the compose
|
|
service and leave an explanatory comment; that is the repo's convention.
|
|
|
|
## Traps that make this hard to see
|
|
|
|
- **Editing a compose service does not replace a running container.**
|
|
`restart: unless-stopped` will happily keep an old container — old image, old
|
|
device list, old mounts — alive forever. One instance ran 9,146 restart
|
|
attempts on stale config before anyone looked. If the container predates the
|
|
edit, recreate it explicitly; do not debug the service.
|
|
- **`podman inspect --format '{{.HostConfig.Devices}}'` is not a device check.**
|
|
It can read empty on a container that demonstrably has the devices. Read the
|
|
real argv with `{{.Config.CreateCommand}}` and, decisively, probe from inside:
|
|
`podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available())'`.
|
|
- **A monitoring target's failure string can prove a fix.** A `/metrics` scrape
|
|
that flips from `connection refused` to `unsupported Content-Type "text/html"`
|
|
means the service came up and only the scrape is wrong. Read the error, not
|
|
just up/down — and never "fix" a red probe by deleting the job, because an
|
|
absent scrape and a healthy one look identical.
|
|
- **A hand-written `~/.config/systemd/user/<svc>.service` that shells out to
|
|
`podman compose up` collides by name with the Quadlet the generator produces
|
|
for `<svc>.container`.** Retire it to a dated trash directory before
|
|
installing the unit.
|
|
- **`is-enabled` saying `generated` is the healthy state** for Quadlet units;
|
|
`systemctl --user enable` fails on them.
|
|
|
|
## Migration shape that works
|
|
|
|
1. Unit file in the repo at `<host>/<service>/quadlet/<service>.container`:
|
|
`Image=` with the `localhost/` prefix for locally built images, `Pull=never`,
|
|
explicit `PublishPort` (loopback unless the service is the documented LAN
|
|
exception), `AddDevice=`/`GroupAdd=keep-groups` where a GPU is needed,
|
|
`Restart=always`, `[Install] WantedBy=default.target`.
|
|
2. A healthcheck **as a file**, never inline: the podman 5.x v1 generator
|
|
splits `HealthCmd` on spaces, so `python3 -c "..."` becomes a quoting puzzle
|
|
that fails silently.
|
|
3. `deploy_<svc>()` in the remote handler: install the unit, retire any
|
|
compose-era unit into `~/.AGENTTRASH/`, `podman rm -f <container>` so the
|
|
unit's own can take the name, daemon-reload, restart, wait for health.
|
|
4. Delete the compose service (one owner per container name) and leave a comment
|
|
naming the unit, so the next reader does not re-add it.
|
|
5. Planner: add a path rule (`<host>/<dir>/**` -> service) **and** a
|
|
`service_is_allowed` entry, or the shared dispatcher rejects the service
|
|
before anything runs.
|
|
6. Migrate the seams: tests that read the compose file for devices, command
|
|
flags or groups must read the unit instead — and resolve file paths from the
|
|
tooling's own declarations, never from `find ... | head -1`.
|
|
|
|
## When the last service leaves a compose file
|
|
|
|
The repo's compose-validity seam requires at least one service, so a file whose
|
|
services have all migrated fails its own test. Either keep the file as
|
|
comments and relax that seam deliberately, or delete the file and fix the
|
|
planner's compose-diff path — decide with the owner, because the deploy tooling
|
|
reads it.
|