Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
78 lines
3.8 KiB
Markdown
78 lines
3.8 KiB
Markdown
---
|
|
name: stale-compose-container-quadlet-migration
|
|
description: "Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit."
|
|
---
|
|
|
|
# Diagnose a crash-looping container after a reboot: stale compose container
|
|
|
|
## When to use
|
|
|
|
A service that worked before a reboot now crash-loops (or is `Exited`), its
|
|
config file looks correct, and the failure message blames something the config
|
|
does declare — e.g. `RuntimeError: No HIP GPUs are available` for a container
|
|
whose compose service passes `/dev/kfd`.
|
|
|
|
## Root-cause pattern
|
|
|
|
The running container was created by an EARLIER revision of the service
|
|
(before the missing lines were added) and `restart: unless-stopped` keeps
|
|
resurrecting it:
|
|
|
|
- `podman compose up -d` does not recreate an existing container because its
|
|
*declaration* changed — it only reconciles what compose tracks.
|
|
- So the container keeps its original flags forever, and every restart mode
|
|
(`unless-stopped`, `always`, podman-compose unit, quadlet) faithfully keeps
|
|
restarting a container that can never work.
|
|
- Symptom shape: high `RestartCount`, exit code 0 or 1, a config that looks
|
|
right, and a failure that contradicts the config.
|
|
|
|
## Confirm in three commands
|
|
|
|
```bash
|
|
# 1. How long has it been looping, and what actually started it?
|
|
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'
|
|
|
|
# 2. The flags it REALLY has — this is ground truth. Do NOT trust
|
|
# {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
|
|
# containers that were started with --device and --group-add.
|
|
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'
|
|
|
|
# 3. Behavioural truth inside it
|
|
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'
|
|
```
|
|
|
|
Compare the container's `created` timestamp with when the config gained the
|
|
missing lines (`git log -p --follow <config>`). Created-before-the-edit is the
|
|
signature.
|
|
|
|
## Fix
|
|
|
|
Recreating the compose container works, but on a repo-governed fleet the fix is
|
|
to give the service a **repo-tracked, boot-persistent Quadlet unit** (every
|
|
container should already be one), then retire the compose path:
|
|
|
|
1. `docker/quadlet/<name>.container` with the full run posture: image (use
|
|
`Image=localhost/<tag>` for locally built images + `Pull=never`),
|
|
`AddDevice=`, `GroupAdd=keep-groups`, `Volume=`, `PublishPort=127.0.0.1:...`
|
|
for an unguarded service, `HealthCmd=`, `Restart=always`, and
|
|
`[Install] WantedBy=default.target`.
|
|
2. Healthcheck as a **mounted file**, not inline: the podman 5.x v1 generator
|
|
splits `HealthCmd` on spaces into an argv array, so
|
|
`python3 -c "..."` becomes a silent quoting puzzle.
|
|
3. Delete the compose service (one owner per container name) and make the
|
|
deployer build → `install_quadlet` → retire any legacy hand-written unit
|
|
into `~/.AGENTTRASH/` → `podman rm -f <name>` (so the unit's container can
|
|
take the name) → restart → `wait_for_health`.
|
|
4. Wire the planner (`<target>/<svc>/**` → service) and the dispatcher
|
|
allowlist (`<target>:<svc>`), or the deploy refuses the service.
|
|
5. Update the seams to read the unit rather than compose for devices, command
|
|
and groups — and resolve build inputs from the deployer's declaration, never
|
|
from a `find | head -1` directory listing.
|
|
|
|
## Verify behaviourally, not structurally
|
|
|
|
`systemctl --user is-enabled` → `generated`, container `healthy`,
|
|
`RestartCount` stops climbing, the port answers (`curl -o /dev/null -w
|
|
'%{http_code}'`), and the in-container probe reports the device
|
|
(`hip available: True | Radeon 8060S Graphics`).
|