Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
83 lines
3.7 KiB
Markdown
83 lines
3.7 KiB
Markdown
---
|
|
name: container-config-drift-crashloop
|
|
description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
|
|
---
|
|
|
|
# A container that restarts forever while its definition is correct
|
|
|
|
## When to use
|
|
|
|
A container is crash-looping or perpetually "unhealthy", but the file that
|
|
defines it (compose service, unit, deploy script) looks right. Also when a
|
|
service has been quietly down for days and nobody noticed, because a restart
|
|
policy kept something running.
|
|
|
|
The cause is usually **config drift**: the running container was created from an
|
|
older definition, and the restart policy restarts it instead of recreating it.
|
|
A compose/unit edit does not touch an existing container.
|
|
|
|
## Diagnose (5 minutes, in this order)
|
|
|
|
1. **Who started it?** The label names the owner, not your guesswork:
|
|
|
|
```
|
|
podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
|
|
```
|
|
|
|
`podman-compose@…` means compose owns it, whatever unit files exist.
|
|
Compare `created` with when the definition last changed. A container older
|
|
than the edit that should have replaced it is the bug.
|
|
|
|
2. **What is it ACTUALLY running?** Read the argv, not the summary fields:
|
|
|
|
```
|
|
podman inspect <name> --format '{{.Config.CreateCommand}}'
|
|
```
|
|
|
|
Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
|
|
flags are present — a false negative that can send you hunting the wrong
|
|
root cause. `CreateCommand` is the truth; grep it for the flag you care
|
|
about (`--device`, `--group-add`, `-e`, mounts).
|
|
|
|
3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
|
|
A capability error (`No HIP GPUs are available`) alongside a missing device
|
|
flag in step 2 confirms drift rather than a code or image fault. Count the
|
|
damage: `podman inspect --format '{{.RestartCount}}'`.
|
|
|
|
## Fix
|
|
|
|
Quickest unblock: recreate from the definition (`podman compose up -d
|
|
--force-recreate <svc>`), which is what should have happened automatically.
|
|
|
|
Durable fix, when the host is governed by a repo that requires boot-persistent
|
|
units (this one does): migrate the service to a Quadlet. Order that worked:
|
|
|
|
1. build the image and `install` the unit file to `~/.config/containers/systemd/`
|
|
2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
|
|
leftover `~/.config/systemd/user/<svc>.service` collides by name with the
|
|
unit the Quadlet generator produces
|
|
3. `podman rm -f <container>` — the stale container holds the name the unit's
|
|
container needs, and it carries the old config
|
|
4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
|
|
5. delete the compose service so only one owner remains per container name
|
|
|
|
## Verify the capability, not the status
|
|
|
|
`Up` and even `healthy` can be true of a half-configured container. Check the
|
|
thing that was broken, from inside:
|
|
|
|
```
|
|
podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
|
|
ss -ltn | grep <port> # confirm the bind addresses, not the intent
|
|
```
|
|
|
|
And confirm boot persistence the way the generator exposes it:
|
|
`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
|
|
explicit `enable` fails on generated units.
|
|
|
|
## Related reading in this repo
|
|
|
|
- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
|
|
- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
|
|
- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice
|