0 Fork 0

Files

T

Frank 9bfa0be85d chore(skills): promote runbook skills to managed-skills; drop superseded SKILL.md

- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap
  runbooks etc.) now tracked.
- Removed superseded skills: omp_harness_config, peon_ping_integration
  (replaced by user-authored peon-ping-* managed skills),
  rocmfp4-strix-benchmark-orchestration.
- New memory-root skills: niri_compositor_ops, omp_git_workflow,
  omp_model_config, quadlet-media-migration, homelab-service-triage,
  hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.

2026-09-09 16:20:28 -07:00

3.1 KiB

Raw Blame History

name, description

name	description
podman-recreate-anon-volume-wipe	Diagnose "container data wiped after restart" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache).

Podman recreation & anonymous-volume wipe audit

Mechanism (verified live 2026-08-31 on Dankclaw)

A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is podman run --name systemd-%N --replace --rm .... Every systemctl --user restart <unit>.service (and every deploy that does one, e.g. deploy_hermes) destroys and recreates the container. Consequences at recreation time:

Writable-layer writes: gone by definition.
Anonymous volumes (image-declared VOLUME with no explicit -v/Volume= for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (podman volume prune only removes unmounted ones).
Named volumes and bind mounts: persist (remounted by name/path).

Audit procedure

ssh <host> 'podman ps -a' — map which host runs which stack (aliases dankclaw/dankserver match their hostnames).
Per container: podman inspect <name> --format "{{json .Mounts}}" → classify each mount "Type":"bind" vs "Type":"volume" (hex-name volume = anonymous).
Quirks: podman ps --format template field is .Names, not .Name (psReporter has no Name field — using .Name errors). Volume _data dirs under ~/.local/share/containers/storage/volumes/<id>/_data are mode 0700 — unreadable by the plain ssh user; use podman unshare/sudo if content inspection is required.

Fleet state at audit time

hermes trio (systemd-hermes, hermes-dashboard, hermes-webui; gateway container auto-names systemd-<unit> because its unit lacks ContainerName=): /data = bind of /home/dank/hermes-data (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; /opt/data (image default VOLUME + HERMES_LAZY_INSTALL_TARGET) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside /data//webui was reset, not the durable state.
romm (dankserver): runs with anonymous volumes at /romm and /redis-data → its DB lives in an anon volume; recreating romm (podman-compose up -d after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
searxng: /var/cache/searxng is an anonymous volume — cache only, benign reset on recreation.
Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).

Fixes when data must survive recreation

Pin a NAMED volume (Volume=hermes-data:/opt/data in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.

3.1 KiB Raw Blame History