agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: stale-compose-container-quadlet-migration
|
||||
description: "Diagnose a container that crash-loops after a reboot because a stale, pre-config-change compose container keeps being restarted, confirm it with the container's real CreateCommand, and migrate the service to a repo-tracked Quadlet unit."
|
||||
---
|
||||
|
||||
# Diagnose a crash-looping container after a reboot: stale compose container
|
||||
|
||||
## When to use
|
||||
|
||||
A service that worked before a reboot now crash-loops (or is `Exited`), its
|
||||
config file looks correct, and the failure message blames something the config
|
||||
does declare — e.g. `RuntimeError: No HIP GPUs are available` for a container
|
||||
whose compose service passes `/dev/kfd`.
|
||||
|
||||
## Root-cause pattern
|
||||
|
||||
The running container was created by an EARLIER revision of the service
|
||||
(before the missing lines were added) and `restart: unless-stopped` keeps
|
||||
resurrecting it:
|
||||
|
||||
- `podman compose up -d` does not recreate an existing container because its
|
||||
*declaration* changed — it only reconciles what compose tracks.
|
||||
- So the container keeps its original flags forever, and every restart mode
|
||||
(`unless-stopped`, `always`, podman-compose unit, quadlet) faithfully keeps
|
||||
restarting a container that can never work.
|
||||
- Symptom shape: high `RestartCount`, exit code 0 or 1, a config that looks
|
||||
right, and a failure that contradicts the config.
|
||||
|
||||
## Confirm in three commands
|
||||
|
||||
```bash
|
||||
# 1. How long has it been looping, and what actually started it?
|
||||
podman inspect <c> --format 'restarts={{.RestartCount}} created={{.Created}} exit={{.State.ExitCode}} unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'
|
||||
|
||||
# 2. The flags it REALLY has — this is ground truth. Do NOT trust
|
||||
# {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}}: both report 0/[] for
|
||||
# containers that were started with --device and --group-add.
|
||||
podman inspect <c> --format '{{.Config.CreateCommand}}' | tr ' ' '\n' | grep -A1 -E 'device|group-add'
|
||||
|
||||
# 3. Behavioural truth inside it
|
||||
podman exec <c> python3 -c 'import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))'
|
||||
```
|
||||
|
||||
Compare the container's `created` timestamp with when the config gained the
|
||||
missing lines (`git log -p --follow <config>`). Created-before-the-edit is the
|
||||
signature.
|
||||
|
||||
## Fix
|
||||
|
||||
Recreating the compose container works, but on a repo-governed fleet the fix is
|
||||
to give the service a **repo-tracked, boot-persistent Quadlet unit** (every
|
||||
container should already be one), then retire the compose path:
|
||||
|
||||
1. `docker/quadlet/<name>.container` with the full run posture: image (use
|
||||
`Image=localhost/<tag>` for locally built images + `Pull=never`),
|
||||
`AddDevice=`, `GroupAdd=keep-groups`, `Volume=`, `PublishPort=127.0.0.1:...`
|
||||
for an unguarded service, `HealthCmd=`, `Restart=always`, and
|
||||
`[Install] WantedBy=default.target`.
|
||||
2. Healthcheck as a **mounted file**, not inline: the podman 5.x v1 generator
|
||||
splits `HealthCmd` on spaces into an argv array, so
|
||||
`python3 -c "..."` becomes a silent quoting puzzle.
|
||||
3. Delete the compose service (one owner per container name) and make the
|
||||
deployer build → `install_quadlet` → retire any legacy hand-written unit
|
||||
into `~/.AGENTTRASH/` → `podman rm -f <name>` (so the unit's container can
|
||||
take the name) → restart → `wait_for_health`.
|
||||
4. Wire the planner (`<target>/<svc>/**` → service) and the dispatcher
|
||||
allowlist (`<target>:<svc>`), or the deploy refuses the service.
|
||||
5. Update the seams to read the unit rather than compose for devices, command
|
||||
and groups — and resolve build inputs from the deployer's declaration, never
|
||||
from a `find | head -1` directory listing.
|
||||
|
||||
## Verify behaviourally, not structurally
|
||||
|
||||
`systemctl --user is-enabled` → `generated`, container `healthy`,
|
||||
`RestartCount` stops climbing, the port answers (`curl -o /dev/null -w
|
||||
'%{http_code}'`), and the in-container probe reports the device
|
||||
(`hip available: True | Radeon 8060S Graphics`).
|
||||
Reference in new issue
Block a user