---
name: podman-recreate-anon-volume-wipe
description: "Diagnose \"container data wiped after restart\" on this rootless-podman fleet: quadlet restarts recreate containers via podman run --replace --rm, and anonymous volumes reset to empty on every recreation. Includes the live audit procedure and the known affected services (hermes trio /opt/data, romm /romm DB, searxng cache)."
---

# Podman recreation & anonymous-volume wipe audit

## Mechanism (verified live 2026-08-31 on Dankclaw)
A quadlet unit restart is NOT a start of the same container: the generated unit's ExecStart is
`podman run --name systemd-%N --replace --rm ...`. Every `systemctl --user restart <unit>.service` (and every deploy that does one, e.g. `deploy_hermes`) **destroys and recreates** the container. Consequences at recreation time:

- Writable-layer writes: gone by definition.
- Anonymous volumes (image-declared `VOLUME` with no explicit `-v`/`Volume=` for that path): NOT carried across recreation — the new container gets a FRESH EMPTY one seeded only from image content. Orphans stay on disk until pruned (`podman volume prune` only removes unmounted ones).
- Named volumes and bind mounts: persist (remounted by name/path).

## Audit procedure
1. `ssh <host> 'podman ps -a'` — map which host runs which stack (aliases `dankclaw`/`dankserver` match their hostnames).
2. Per container: `podman inspect <name> --format "{{json .Mounts}}"` → classify each mount `"Type":"bind"` vs `"Type":"volume"` (hex-name volume = anonymous).
3. Quirks: `podman ps --format` template field is `.Names`, not `.Name` (psReporter has no `Name` field — using `.Name` errors). Volume `_data` dirs under `~/.local/share/containers/storage/volumes/<id>/_data` are mode 0700 — unreadable by the plain ssh user; use `podman unshare`/sudo if content inspection is required.

## Fleet state at audit time
- **hermes trio** (`systemd-hermes`, `hermes-dashboard`, `hermes-webui`; gateway container auto-names `systemd-<unit>` because its unit lacks `ContainerName=`): `/data` = bind of `/home/dank/hermes-data` (durable state: config.yaml, sessions/, skills/, state.db) survives recreations; `/opt/data` (image default VOLUME + `HERMES_LAZY_INSTALL_TARGET`) is an anonymous volume → resets to image defaults on every deploy. So "Hermes lost data after restart" = anything written outside `/data`/`/webui` was reset, not the durable state.
- **romm** (dankserver): runs with anonymous volumes at `/romm` and `/redis-data` → its DB lives in an anon volume; recreating romm (`podman-compose up -d` after image bump) boots it with a fresh empty DB. Fix = named volume or bind before next recreation.
- **searxng**: `/var/cache/searxng` is an anonymous volume — cache only, benign reset on recreation.
- Everything else on both hosts was verified as bind mounts (or stateless by design: authentik-server has no mounts at all; anubis is a memory store with client-side JWTs).

## Fixes when data must survive recreation
Pin a NAMED volume (`Volume=hermes-data:/opt/data` in the quadlet unit) or point the env var at a bind-mounted path. Never rely on anonymous volumes surviving recreation.