Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
29 lines
3.1 KiB
Markdown
29 lines
3.1 KiB
Markdown
---
|
|
name: podman-recreate-anon-volume-wipe
|
|
description: "Diagnose \"container data wiped after restart\" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache)."
|
|
---
|
|
|
|
# Podman recreation & anonymous-volume wipe audit
|
|
|
|
## Mechanism (verified live 2026-08-31 on Dankclaw)
|
|
A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is
|
|
`podman run --name systemd-%N --replace --rm ...`. Every `systemctl --user restart <unit>.service` (and every deploy that does one, e.g. `deploy_hermes`) **destroys and recreates** the container. Consequences at recreation time:
|
|
|
|
- Writable-layer writes: gone by definition.
|
|
- Anonymous volumes (image-declared `VOLUME` with no explicit `-v`/`Volume=` for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (`podman volume prune` only removes unmounted ones).
|
|
- Named volumes and bind mounts: persist (remounted by name/path).
|
|
|
|
## Audit procedure
|
|
1. `ssh <host> 'podman ps -a'` — map which host runs which stack (aliases `dankclaw`/`dankserver` match their hostnames).
|
|
2. Per container: `podman inspect <name> --format "{{json .Mounts}}"` → classify each mount `"Type":"bind"` vs `"Type":"volume"` (hex-name volume = anonymous).
|
|
3. Quirks: `podman ps --format` template field is `.Names`, not `.Name` (psReporter has no `Name` field — using `.Name` errors). Volume `_data` dirs under `~/.local/share/containers/storage/volumes/<id>/_data` are mode 0700 — unreadable by the plain ssh user; use `podman unshare`/sudo if content inspection is required.
|
|
|
|
## Fleet state at audit time
|
|
- **hermes trio** (`systemd-hermes`, `hermes-dashboard`, `hermes-webui`; gateway container auto-names `systemd-<unit>` because its unit lacks `ContainerName=`): `/data` = bind of `/home/dank/hermes-data` (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; `/opt/data` (image default VOLUME + `HERMES_LAZY_INSTALL_TARGET`) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside `/data`/`/webui` was reset, not the durable state.
|
|
- **romm** (dankserver): runs with anonymous volumes at `/romm` and `/redis-data` → its DB lives in an anon volume; recreating romm (`podman-compose up -d` after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
|
|
- **searxng**: `/var/cache/searxng` is an anonymous volume — cache only, benign reset on recreation.
|
|
- Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).
|
|
|
|
## Fixes when data must survive recreation
|
|
Pin a NAMED volume (`Volume=hermes-data:/opt/data` in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.
|