agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: container-config-drift-crashloop
|
||||
description: "Diagnose a container that crash-loops or stays unhealthy while its compose/unit definition looks correct: config drift kept alive by a restart policy, with the inspect commands, the Devices-vs-CreateCommand false negative, and the compose-to-Quadlet migration order."
|
||||
---
|
||||
|
||||
# A container that restarts forever while its definition is correct
|
||||
|
||||
## When to use
|
||||
|
||||
A container is crash-looping or perpetually "unhealthy", but the file that
|
||||
defines it (compose service, unit, deploy script) looks right. Also when a
|
||||
service has been quietly down for days and nobody noticed, because a restart
|
||||
policy kept something running.
|
||||
|
||||
The cause is usually **config drift**: the running container was created from an
|
||||
older definition, and the restart policy restarts it instead of recreating it.
|
||||
A compose/unit edit does not touch an existing container.
|
||||
|
||||
## Diagnose (5 minutes, in this order)
|
||||
|
||||
1. **Who started it?** The label names the owner, not your guesswork:
|
||||
|
||||
```
|
||||
podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'
|
||||
```
|
||||
|
||||
`podman-compose@…` means compose owns it, whatever unit files exist.
|
||||
Compare `created` with when the definition last changed. A container older
|
||||
than the edit that should have replaced it is the bug.
|
||||
|
||||
2. **What is it ACTUALLY running?** Read the argv, not the summary fields:
|
||||
|
||||
```
|
||||
podman inspect <name> --format '{{.Config.CreateCommand}}'
|
||||
```
|
||||
|
||||
Trap: `{{.HostConfig.Devices}}` prints an empty list even when `--device`
|
||||
flags are present — a false negative that can send you hunting the wrong
|
||||
root cause. `CreateCommand` is the truth; grep it for the flag you care
|
||||
about (`--device`, `--group-add`, `-e`, mounts).
|
||||
|
||||
3. **What does the process say?** `podman logs <name> | grep -iE 'error|cannot|Runtime'`.
|
||||
A capability error (`No HIP GPUs are available`) alongside a missing device
|
||||
flag in step 2 confirms drift rather than a code or image fault. Count the
|
||||
damage: `podman inspect --format '{{.RestartCount}}'`.
|
||||
|
||||
## Fix
|
||||
|
||||
Quickest unblock: recreate from the definition (`podman compose up -d
|
||||
--force-recreate <svc>`), which is what should have happened automatically.
|
||||
|
||||
Durable fix, when the host is governed by a repo that requires boot-persistent
|
||||
units (this one does): migrate the service to a Quadlet. Order that worked:
|
||||
|
||||
1. build the image and `install` the unit file to `~/.config/containers/systemd/`
|
||||
2. stop/disable the legacy unit and **move its file to `~/.AGENTTRASH/`** — a
|
||||
leftover `~/.config/systemd/user/<svc>.service` collides by name with the
|
||||
unit the Quadlet generator produces
|
||||
3. `podman rm -f <container>` — the stale container holds the name the unit's
|
||||
container needs, and it carries the old config
|
||||
4. `systemctl --user daemon-reload && systemctl --user restart <svc>.service`
|
||||
5. delete the compose service so only one owner remains per container name
|
||||
|
||||
## Verify the capability, not the status
|
||||
|
||||
`Up` and even `healthy` can be true of a half-configured container. Check the
|
||||
thing that was broken, from inside:
|
||||
|
||||
```
|
||||
podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"
|
||||
ss -ltn | grep <port> # confirm the bind addresses, not the intent
|
||||
```
|
||||
|
||||
And confirm boot persistence the way the generator exposes it:
|
||||
`systemctl --user is-enabled <svc>.service` → `generated` is healthy; an
|
||||
explicit `enable` fails on generated units.
|
||||
|
||||
## Related reading in this repo
|
||||
|
||||
- AGENTS.md "Boot Persistence & Self-Healing" — the rule this enforces
|
||||
- `serverAppConfigs-docs/Dankclaw/ComfyUI.md` — the 9,146-restart incident
|
||||
- `serverAppConfigs-docs/Infrastructure/Observability.md` — same class, twice
|
||||
Reference in new issue
Block a user