Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
65 lines
3.7 KiB
Markdown
65 lines
3.7 KiB
Markdown
---
|
|
name: compose-stack-bringup-groundtruth
|
|
description: "Bring up a compose-defined container stack on a rootless podman host by verifying reality first (mounts, images, in-image interfaces) instead of trusting compose env/flag guesses; includes the copyparty ac-edition argv-interface recipe and boot-persistence audit."
|
|
---
|
|
|
|
# Compose-stack bring-up with ground truth (rootless podman hosts)
|
|
|
|
Use when standing up (or debugging) a compose-defined service stack on a rootless
|
|
podman host where the stack has never run, or answers with surprise statuses
|
|
(SIGSEGV at boot, HTTP 500 on `/`, "manifest unknown").
|
|
|
|
## Procedure
|
|
|
|
1. **Audit reality before pressing buttons.** `podman ps -a` runs per-user —
|
|
check every namespace (`sudo podman ps -a`) and native units
|
|
(`systemctl list-units`) before declaring a service absent; a service may run
|
|
as a native systemd unit or snap, not a container. Compare compose mount source
|
|
paths against the real filesystem (`ls /mnt/...`): compose files often name
|
|
phantom paths that never existed (e.g. `/mnt/media/*` vs the real LUKS array at
|
|
`/mnt/entropy`). Fix compose through git first, deploy second.
|
|
|
|
2. **Pre-create bind-mount sources.** Rootless podman fails container creation
|
|
with `statfs /opt/<svc>/...: no such file or directory` when a bind source is
|
|
missing. `sudo mkdir -p` + `chown 1000:1000` every compose volume source
|
|
before the first `podman-compose up -d`.
|
|
|
|
3. **Staged bring-up.** Infra first (`up -d mariadb redis`), poll healthchecks via
|
|
`podman inspect --format "{{.State.Health.Status}}"`, then `up -d` the rest.
|
|
|
|
4. **Never guess image tags — query ground truth.** ghcr.io answers **403 on the
|
|
bearer-token request for nonexistent/private repos** ("manifest unknown" / token
|
|
len 0 = repo or tag does not exist, not an auth problem). Query the registry
|
|
API (`ghcr.io/token?scope=repository:<ns>/<repo>:pull` then
|
|
`/v2/<ns>/<repo>/tags/list`) before pinning a tag.
|
|
|
|
5. **Read the interface from inside the image** before guessing flags/envs:
|
|
`podman exec <ctr> cat /z/cpp.sh`, `cat /z/initcfg`,
|
|
`podman exec <ctr> python3 -m copyparty --help-accounts`. A service answering
|
|
HTTP 500 with its own styled error page is *running* — it is misconfigured,
|
|
not down. Copyparty ac edition ground truth: entrypoint
|
|
`/bin/ash /z/cpp.sh -c /z/initcfg` then `exec python3 -m copyparty "$@"`;
|
|
accounts/volumes are **argv flags** (`-a USER:PASS`, `-v SRC:DST:PERMS`), not
|
|
`COPAY_*` envs (zero `COPAY_` references in the package). With no accounts and
|
|
no volumes configured it serves cwd read/write for everyone — an
|
|
unauthenticated world-writable share of the whole mount.
|
|
|
|
6. **Beware remote-shell quoting traps.** Remote zsh eats bare `===`/`?` tokens
|
|
(glob qualifiers/globs); grep patterns with nested quotes over ssh can be
|
|
quote-eaten into silent zero-match. A zero-match from a mangled pattern is not
|
|
absence — re-verify the target exists before concluding it.
|
|
|
|
7. **Boot-persistence audit last.** On rootless podman hosts with
|
|
`podman-user-generator`, containers started via podman-compose get generated
|
|
`container-<name>.service` units at first boot (evidence: an existing unit for
|
|
one container proves the generator works). Verify after first reboot:
|
|
`systemctl --user is-enabled <name>.service`.
|
|
|
|
## Gotchas
|
|
- jellyfinetype SIGSEGV at boot with `user: "1000:1000"` = unmapped uid inside the
|
|
rootless userns (host uid 1000 == container uid 0); drop the user line and let it
|
|
run as container-root (= host deploy user).
|
|
- `up -d --force-recreate` applies compose changes that change command/env; probe with
|
|
a poll loop (`for i in $(seq...)`) because fresh containers answer `000`/refused for
|
|
the first seconds while indexing.
|