---
name: fleet-maintenance-window-refresh
description: "Execute a fleet-wide OS + container image refresh on Dankclaw/Dankserver: apt without reboots, registry pin census, digest bump to main, per-service deploys, and proving the container actually moved."
---

"Update the servers / update all containers" on Dankclaw (192.168.0.143) + Dankserver (192.168.0.235). Both are Ubuntu 26.04, rootless podman quadlets, deploy via `scripts/deploy.sh`.

## 0. Recon before touching anything

```bash
for h in .143 .235; do ssh dank@192.168.0.$h 'uptime; systemctl --failed --plain --no-legend; sudo -n apt-get -s upgrade | grep -c ^Inst'; done
ssh dank@192.168.0.235 'ps -eo pid,etimes,args | grep -E "radiance|llama-server" | grep -v grep'
```
- A radiance/llama process holding the GPUs is a campaign or a bench: **do not touch it**, and never reboot without explicit approval.
- Note pre-existing failed units so you do not blame them on your deploy later (`mnt-md0.mount`, `nvidia-cdi-refresh*`, `logrotate` fail on Dankserver by default).

## 1. OS packages, no reboot

```bash
sudo -n env DEBIAN_FRONTEND=noninteractive NOTIFY_SOCKET= apt-get update -qq
sudo -n env DEBIAN_FRONTEND=noninteractive NOTIFY_SOCKET= apt-get -y \
  -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold \
  -o APT::Get::AutomaticRemove=false full-upgrade
```
- `needrestart` is NOT installed on these hosts — check before assuming daemons will not bounce.
- Dankserver's initramfs trigger takes ~9 minutes; "dpkg BUSY" for 20 minutes is normal, do not kill it.
- After: `cat /var/run/reboot-required`, `uname -r` vs newest `/boot/vmlinuz-*`. A staged kernel is not a running kernel — say so.
- **Dankclaw only:** `grep -o 'iommu=pt' /boot/grub/grub.cfg` must still match after the kernel's `update-grub`, else the XDNA2 NPU lane dies at the next boot.

## 2. Registry pin census (full digests, not prefixes)

Read every `Image=` in `**/quadlet/*.container` plus `FRANK_*_IMAGE=` in `scripts/*.env`, resolve `tag -> Docker-Content-Digest` and compare.

- Docker Hub: token from `auth.docker.io/token?service=registry.docker.io&scope=repository:<repo>:pull`, then `HEAD registry-1.docker.io/v2/<repo>/manifests/<tag>` with an index+list+manifest `Accept`.
- ghcr.io: `https://ghcr.io/token?scope=repository:<repo>:pull&service=ghcr.io`.
- **lscr.io answers 404 through its own `/v2` path.** Its images are byte-identical on `docker.io/library/linuxserver/<repo>` — enumerate there; pinned lscr digests match Hub digests exactly.
- Print the FULL digest. A 12-char prefix cannot be pasted into a pin.
- `latest`-tagged services differ constantly; a differing digest IS the update. Do not call a versioned pin stale without a newer version tag.

## 3. Bump the pins in the repo

- Digest-only change: `sed -i 's|sha256:OLD|sha256:NEW|g'` (verify with `git diff -U0`).
- Tag+digest change: edit the line.
- Some pins MUST stay tagless `repo@sha256:…` — a bats seam greps the exact form (e.g. `frank-dragoon.env`). Check `grep -n 'FRANK_DRAGOON_IMAGE=' tests/*.bats` before adding a tag.
- Repo-wide truth is the canonical checkout **`/home/blank/Projects/serverAppConfig`** (on `main`, has `tests/e2e/.env.local`). Session worktrees cannot run `deploy.sh`: it refuses a dirty tree and any branch but `main`, and `git checkout` is blocked by workspace policy. Edit in the worktree, then there: `git pull --ff-only`, `git merge --no-ff <branch>`, `git push`.
- Restore canonical-checkout files you edited by accident with `git show HEAD:<path> > <path>` (checkout/restore are blocked).

## 4. Pre-pull, then deploy

Quadlets ship `Pull=never`/`missing` and the deployers refuse an absent image, so pull first as the **owner** (rootless user `dank`):

```bash
ssh dank@host podman pull -q 'repo/name:tag@sha256:...'   # exact pinned ref
```
Then, from the canonical checkout, batch services per host in one invocation (each invocation also runs the ~1 min Playwright gate):

```bash
./scripts/deploy.sh dankclaw --service grafana --service alertmanager --service seerr
```
Order within a batch: DB before app (postgres/valkey/mariadb first), `gluetun` before `searxng` (shared netns). A failure aborts the rest of the batch, so group tightly.

## 5. PROVE the container moved

A green deploy is not evidence. Compare running digests to the pins:

```bash
ssh dank@host podman ps --format '{{.Names}}|{{.Image}}'
```
- Handlers that call `systemctl --user start` no-op on an active unit: the image installs, the old container keeps serving, and the health gate + e2e still pass. (This bit authentik server/worker on 2026-10-30; the handler now restarts.) If a digest did not move, `systemctl --user restart <unit>` and re-check.
- `systemctl is-active` without `--user` reports system units — Dankclaw app quadlets are user units. Always `--user` for them.

## 6. Locally built images (loki, alloy, caddy-porkbun)

`build_compose_image` skips the build when the image's `cfhash` label matches the Containerfile hash, so a rolling base (`alpine:latest`, `grafana/loki:latest`, `caddy:2`) never refreshes on its own. Force it by rebuilding with the same label, then deploy to restart:

```bash
h=$(sha256sum dankclaw/loki/Containerfile | cut -d' ' -f1)
podman build --format docker --no-cache --label "cfhash=$h" -t franksdank/${svc}:latest -f <Containerfile> .
```
- **zsh trap:** over `ssh` the remote shell is zsh, where `$svc:latest` is a history modifier (`:l` lowercases) — `-t franksdank/$svc:latest` tags `lokiatest`. Always brace: `${svc}:latest`.
- alloy is mirrored to the ROOT store by its handler (`mirror_image_to_root`), so deploy it rather than restarting the user unit.

## 7. Close

`./tests/run.sh` before each push (577 seams, ~85 s). Docs: update the endpoint/service page as the owner of the fact; `AGENTS.md` only for a cross-cutting law. Report the reboot as OWED, not done, when it is staged.