command-center/agent/managed-skills/fleet-maintenance-window-refresh/SKILL.md

name, description

name	description
fleet-maintenance-window-refresh	Execute a fleet-wide OS + container image refresh on Dankclaw/Dankserver: apt without reboots, registry pin census, digest bump to main, per-service deploys, and proving the container actually moved.

"Update the servers / update all containers" on Dankclaw (192.168.0.143) + Dankserver (192.168.0.235). Both are Ubuntu 26.04, rootless podman quadlets, deploy via scripts/deploy.sh.

0. Recon before touching anything

for h in .143 .235; do ssh dank@192.168.0.$h 'uptime; systemctl --failed --plain --no-legend; sudo -n apt-get -s upgrade | grep -c ^Inst'; done
ssh dank@192.168.0.235 'ps -eo pid,etimes,args | grep -E "radiance|llama-server" | grep -v grep'

• A radiance/llama process holding the GPUs is a campaign or a bench: do not touch it, and never reboot without explicit approval.
• Note pre-existing failed units so you do not blame them on your deploy later (mnt-md0.mount, nvidia-cdi-refresh*, logrotate fail on Dankserver by default).

1. OS packages, no reboot

sudo -n env DEBIAN_FRONTEND=noninteractive NOTIFY_SOCKET= apt-get update -qq
sudo -n env DEBIAN_FRONTEND=noninteractive NOTIFY_SOCKET= apt-get -y \
  -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold \
  -o APT::Get::AutomaticRemove=false full-upgrade

• needrestart is NOT installed on these hosts — check before assuming daemons will not bounce.
• Dankserver's initramfs trigger takes ~9 minutes; "dpkg BUSY" for 20 minutes is normal, do not kill it.
• After: cat /var/run/reboot-required, uname -r vs newest /boot/vmlinuz-*. A staged kernel is not a running kernel — say so.
• Dankclaw only: grep -o 'iommu=pt' /boot/grub/grub.cfg must still match after the kernel's update-grub, else the XDNA2 NPU lane dies at the next boot.

2. Registry pin census (full digests, not prefixes)

Read every Image= in **/quadlet/*.container plus FRANK_*_IMAGE= in scripts/*.env, resolve tag -> Docker-Content-Digest and compare.

• Docker Hub: token from auth.docker.io/token?service=registry.docker.io&scope=repository:<repo>:pull, then HEAD registry-1.docker.io/v2/<repo>/manifests/<tag> with an index+list+manifest Accept.
• ghcr.io: https://ghcr.io/token?scope=repository:<repo>:pull&service=ghcr.io.
• lscr.io answers 404 through its own /v2 path. Its images are byte-identical on docker.io/library/linuxserver/<repo> — enumerate there; pinned lscr digests match Hub digests exactly.
• Print the FULL digest. A 12-char prefix cannot be pasted into a pin.
• latest-tagged services differ constantly; a differing digest IS the update. Do not call a versioned pin stale without a newer version tag.

3. Bump the pins in the repo

• Digest-only change: sed -i 's|sha256:OLD|sha256:NEW|g' (verify with git diff -U0).
• Tag+digest change: edit the line.
• Some pins MUST stay tagless repo@sha256:… — a bats seam greps the exact form (e.g. frank-dragoon.env). Check grep -n 'FRANK_DRAGOON_IMAGE=' tests/*.bats before adding a tag.
• Repo-wide truth is the canonical checkout /home/blank/Projects/serverAppConfig (on main, has tests/e2e/.env.local). Session worktrees cannot run deploy.sh: it refuses a dirty tree and any branch but main, and git checkout is blocked by workspace policy. Edit in the worktree, then there: git pull --ff-only, git merge --no-ff <branch>, git push.
• Restore canonical-checkout files you edited by accident with git show HEAD:<path> > <path> (checkout/restore are blocked).

4. Pre-pull, then deploy

Quadlets ship Pull=never/missing and the deployers refuse an absent image, so pull first as the owner (rootless user dank):

ssh dank@host podman pull -q 'repo/name:tag@sha256:...'   # exact pinned ref

Then, from the canonical checkout, batch services per host in one invocation (each invocation also runs the ~1 min Playwright gate):

./scripts/deploy.sh dankclaw --service grafana --service alertmanager --service seerr

Order within a batch: DB before app (postgres/valkey/mariadb first), gluetun before searxng (shared netns). A failure aborts the rest of the batch, so group tightly.

5. PROVE the container moved

A green deploy is not evidence. Compare running digests to the pins:

ssh dank@host podman ps --format '{{.Names}}|{{.Image}}'

• Handlers that call systemctl --user start no-op on an active unit: the image installs, the old container keeps serving, and the health gate + e2e still pass. (This bit authentik server/worker on 2026-10-30; the handler now restarts.) If a digest did not move, systemctl --user restart <unit> and re-check.
• systemctl is-active without --user reports system units — Dankclaw app quadlets are user units. Always --user for them.

6. Locally built images (loki, alloy, caddy-porkbun)

build_compose_image skips the build when the image's cfhash label matches the Containerfile hash, so a rolling base (alpine:latest, grafana/loki:latest, caddy:2) never refreshes on its own. Force it by rebuilding with the same label, then deploy to restart:

h=$(sha256sum dankclaw/loki/Containerfile | cut -d' ' -f1)
podman build --format docker --no-cache --label "cfhash=$h" -t franksdank/${svc}:latest -f <Containerfile> .

• zsh trap: over ssh the remote shell is zsh, where $svc:latest is a history modifier (:l lowercases) — -t franksdank/$svc:latest tags lokiatest. Always brace: ${svc}:latest.
• alloy is mirrored to the ROOT store by its handler (mirror_image_to_root), so deploy it rather than restarting the user unit.

7. Close

./tests/run.sh before each push (577 seams, ~85 s). Docs: update the endpoint/service page as the owner of the fact; AGENTS.md only for a cross-cutting law. Report the reboot as OWED, not done, when it is staged.