docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank
@@ -0,0 +1,151 @@
|
||||
---
|
||||
type: reference
|
||||
title: Deploy State Triage
|
||||
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
|
||||
tags: [deployment, deploy-state, systemd, podman, triage]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Deploy State Triage
|
||||
|
||||
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
on each target (rootless, user `dank`, format exactly two lines
|
||||
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
|
||||
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
|
||||
|
||||
## 1. Stale marker vs live container
|
||||
|
||||
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
|
||||
a restart after the checkout ff-pulled to a newer revision — starts the
|
||||
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
|
||||
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
||||
"behavior pending".
|
||||
|
||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||
possibly zero behavioral change.
|
||||
|
||||
**Procedure (read-only first):**
|
||||
|
||||
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
|
||||
operational delta is usually one `scripts/<svc>.env`.
|
||||
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
||||
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
||||
- `systemctl is-active/is-enabled <svc>`;
|
||||
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
||||
- the on-host env file already carries the new keys (it landed via the
|
||||
ff-pull).
|
||||
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
|
||||
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
|
||||
`sudo podman` inspects an empty store):
|
||||
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
|
||||
the pull ⇒ it read the new env
|
||||
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
||||
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
||||
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
||||
- confirm the launcher maps each tunable to its flag
|
||||
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
|
||||
--enable-prefix-caching`.
|
||||
|
||||
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
||||
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
||||
container lacks the flag/env (or predates the pull) is a real
|
||||
`deploy.sh <target> --service <svc>` warranted.
|
||||
|
||||
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
|
||||
|
||||
```bash
|
||||
S=$HOME/.local/state/serverAppConfigs/deploy-state
|
||||
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
|
||||
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
|
||||
```
|
||||
|
||||
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
|
||||
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
|
||||
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
|
||||
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
|
||||
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
|
||||
no-op).
|
||||
|
||||
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
|
||||
alone — both are bookkeeping; the container's argv/env is ground truth.
|
||||
Reading container state with `sudo podman` / bare `docker`. Restarting a
|
||||
healthy heavy endpoint to satisfy a stale marker.
|
||||
|
||||
## 2. Pending lockout
|
||||
|
||||
Symptom: `deploy.sh <target>` fails or silently stops with
|
||||
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
|
||||
appears.
|
||||
|
||||
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
|
||||
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
|
||||
`echo $?`.
|
||||
2. Inspect the state store (no sudo):
|
||||
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
|
||||
A `status=pending` entry for a service absent from `service_is_allowed`
|
||||
in `scripts/deploy-remote.sh` = retired-service lockout.
|
||||
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
|
||||
not-allowed pending entries to
|
||||
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
|
||||
continues. Verify the WARN line names the service and the state file is
|
||||
gone.
|
||||
4. On a host without the fix: move the stale entry to
|
||||
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
|
||||
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
|
||||
`deploy-state/` no longer holds the retired name.
|
||||
|
||||
Related contracts: the superseded seam
|
||||
(`deployment_scoping.bats` "remote runner recognizes a pending service at
|
||||
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
|
||||
`deployment_flow.bats` extracts `add_pending_services` via sed and
|
||||
HOME-overrides the trash path. **When retiring a service going forward:**
|
||||
delete or reconcile its deploy-state entry as part of the retirement commit —
|
||||
the prune is a safety net, not the intended cleanup path.
|
||||
|
||||
## 3. Interrupted or raced deploys
|
||||
|
||||
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
|
||||
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
|
||||
host-side one** — two `deploy-remote.sh` processes can end up managing the
|
||||
same unit, one deploying a revision you already superseded. Check:
|
||||
`pgrep -af "deploy-remote"` on the target, read the revision argument on
|
||||
each, kill stale ones by explicit PID.
|
||||
|
||||
A killed deploy leaves behind:
|
||||
|
||||
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
||||
explicit stop — start it yourself);
|
||||
- the state marker **pending** (pre-deploy or in-flight revision);
|
||||
- the e2e gate un-run (the deployer runs it last).
|
||||
|
||||
Reconcile the marker ONLY after the service itself is verified healthy — a
|
||||
marker that runs ahead of reality is worse than one that lags (the planner
|
||||
uses it to decide what is pending). Then close the loop:
|
||||
|
||||
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
|
||||
slots, argv flags, `NRestarts`), not just health 200.
|
||||
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
|
||||
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
|
||||
does NOT have it; copy it in first).
|
||||
3. If the killed revision was restoring a known-good shape, confirm the
|
||||
numbers match the last measured-good boot of that shape — equality is the
|
||||
proof, not health alone.
|
||||
|
||||
Boot failures must be read from the journal, not `podman logs`: serve scripts
|
||||
`podman rm -f` the container on every start (previous attempt lost), and
|
||||
`podman logs` on a container still *Stopping* returns the OLD configuration's
|
||||
numbers — which looks exactly like your change having no effect. Key on
|
||||
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
||||
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
||||
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
|
||||
procedure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deploy pipeline itself
|
||||
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
type: reference
|
||||
title: Quadlet Lifecycle
|
||||
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
|
||||
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Quadlet Lifecycle
|
||||
|
||||
Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed,
|
||||
working container MUST have a self-healing, boot-persistent Quadlet unit —
|
||||
`.container` file in the repo (`<host>/<svc>/quadlet/`), installed by the
|
||||
deployer, `[Install] WantedBy=default.target`, `Restart=always`, linger on.
|
||||
A container that runs only from compose dies silently at reboot. This page
|
||||
is the operating manual for that lifecycle.
|
||||
|
||||
## 1. Ownership audit — find the containers that will not survive a reboot
|
||||
|
||||
```bash
|
||||
for c in $(podman ps --format '{{.Names}}'); do
|
||||
printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
|
||||
done
|
||||
```
|
||||
|
||||
- **label present** → the unit owns it; persistence = `[Install]` + `loginctl
|
||||
show-user <user> -p Linger` = yes.
|
||||
- **label absent, compose service exists** → the classic gap: dies at reboot.
|
||||
Check the publish address while there — compose forms often bind the
|
||||
wildcard (an unauthenticated UI/API on every interface).
|
||||
- **label present AND a compose service with the same name** → two owners; a
|
||||
stray `podman compose up` can create a duplicate that fights the unit for
|
||||
the name and the port. Remove the compose service, leave an explanatory
|
||||
comment (repo convention).
|
||||
|
||||
## 2. Cutover: compose service → Quadlet unit
|
||||
|
||||
Checklist (precedents to copy:
|
||||
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
|
||||
`dankclaw/observability/quadlet/`):
|
||||
|
||||
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
|
||||
line for line: `Image=` (`localhost/` prefix for locally built images +
|
||||
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
|
||||
containers by name), `Network=`, `User=0:0` when the image's own user
|
||||
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
|
||||
`Exec=` for compose `command:`, `PublishPort=`, health keys,
|
||||
`[Service] Restart=always` + `RestartSec=10`, `[Install]
|
||||
WantedBy=default.target`. **Never `systemctl --user enable` the generated
|
||||
unit** — `[Install]` + linger IS the persistence (`is-enabled` = `generated`
|
||||
is healthy; explicit enable fails with "transient or generated").
|
||||
2. Exposure: publish on `127.0.0.1` unless something off-host genuinely needs
|
||||
it (Caddy reaches containers by name over the shared network; the wildcard
|
||||
publish is a guard-chain bypass).
|
||||
3. Handler in `scripts/deploy-remote.sh`: `install_quadlet <dir> <name>` →
|
||||
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
|
||||
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
|
||||
`~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides BY NAME with the generated unit);
|
||||
`podman rm -f <container>` so the unit's container can take the name.
|
||||
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
|
||||
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
|
||||
ordered. Add a `test-changed` case above the generic
|
||||
`*/quadlet/*.container` fallback.
|
||||
5. Retire the compose entry in the SAME change — two owners is how a stale
|
||||
duplicate comes back.
|
||||
6. Seam test `tests/<svc>_quadlet.bats`: boot persistence, restart policy, no
|
||||
wildcard publish, container name matches Caddy/scrape references, health
|
||||
command, deployer installs+restarts+waits, never `rm`, never `enable`.
|
||||
Migrate any older assertion that read the compose file for devices/flags.
|
||||
7. Deploy `--service <name>`; prove: `is-active`/`is-enabled` =
|
||||
active/generated, container `healthy`, endpoint answers,
|
||||
`list-dependencies default.target` shows the unit, off-host probes on
|
||||
previously published ports now refuse, and the container no longer carries
|
||||
`io.podman.compose.project`.
|
||||
|
||||
**Traps that cost real time:**
|
||||
|
||||
- `HealthCmd` = CMD-SHELL (podman runs `/bin/sh -c`); a distroless image has
|
||||
no shell → the check fails forever. `--health-cmd` won't take exec form
|
||||
either. Put `HEALTHCHECK … CMD ["binary","arg"]` in the image and build
|
||||
with `--format docker` — the default OCI format has no HEALTHCHECK field
|
||||
and drops it with only a warning.
|
||||
- Never point a unit's `EnvironmentFile` at the repo-root `.env` — it is a
|
||||
gluetun env carrying `HOME=/root`; systemd sets HOME for the service and
|
||||
the unit dies with `stat /root/.config: permission denied`. Use a
|
||||
per-service env under `/home/dank/Containers/<svc>/env`, seeded by the
|
||||
handler with a single-variable filter, validated so a missing secret fails
|
||||
loudly.
|
||||
- `Network=host` exporters scraped by containerized Prometheus must bind the
|
||||
host's LAN address (containers cannot reach host loopback) PLUS `127.0.0.1`
|
||||
for the homepage probe — two `--web.listen-address` flags, never the
|
||||
wildcard.
|
||||
- alloy 1.x refuses the `ui` config block; the unit must pass
|
||||
`--server.http.listen-addr=0.0.0.0:12345` before its port can be published.
|
||||
- GPU services: `AddDevice=` + `GroupAdd=keep-groups` (user-unit spawns carry
|
||||
no supplementary groups).
|
||||
- When the last service leaves a compose file, the compose-validity seam
|
||||
fails (requires ≥1 service): keep the file as comments and relax the seam
|
||||
deliberately, or delete the file and fix the planner's compose-diff path —
|
||||
decide with the owner, the deploy tooling reads it.
|
||||
|
||||
## 3. Drift / crash-loop diagnosis (5 minutes, in order)
|
||||
|
||||
Symptom: crash-looping or perpetually unhealthy container whose DEFINITION
|
||||
looks right; or a service quietly down for days because a restart policy kept
|
||||
something running. Cause: the running container was created from an OLDER
|
||||
definition — a compose/unit edit does not touch an existing container, and
|
||||
the restart policy faithfully resurrects it forever.
|
||||
|
||||
1. **Who started it:** `podman inspect <name> --format 'unit={{index
|
||||
.Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}}
|
||||
created={{.Created}}'`; compare `created` with when the definition changed
|
||||
(`git log -p --follow <config>`). Created-before-the-edit is the signature.
|
||||
2. **What it is ACTUALLY running:** `{{.Config.CreateCommand}}` is the truth.
|
||||
`{{.HostConfig.Devices}}` / `{{.HostConfig.GroupAdd}}` print empty/0 even
|
||||
for containers started with `--device`/`--group-add` — a false negative
|
||||
that sends you hunting the wrong root cause.
|
||||
3. **What the process says:** `podman logs <name> | grep -iE 'error|cannot|Runtime'`
|
||||
(a capability error like `No HIP GPUs are available` beside a missing
|
||||
device flag confirms drift). Count the damage: `{{.RestartCount}}` — one
|
||||
instance reached 9,146 restart attempts on stale config.
|
||||
|
||||
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
= the §2 cutover. **Verify the capability, not the status:** `podman exec
|
||||
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
|
||||
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
|
||||
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
|
||||
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
|
||||
|
||||
## 4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)
|
||||
|
||||
- **Hard-fail keys** — the whole unit silently fails to generate
|
||||
(`systemctl --user cat <unit>` shows "could not be found"): any
|
||||
dash-prefixed `[Container]` option (`--shm-size=…` → `unsupported key …
|
||||
in group 'Container'`); `Command=` is unsupported, use `Exec=`.
|
||||
- **HealthCmd space-splitting:** the v1 generator splits `HealthCmd` on
|
||||
spaces into the healthcheck argv; a quoted probe
|
||||
`/bin/sh -c "pg_isready …"` lands as argv token `"pg_isready` →
|
||||
command-not-found → container reads **unhealthy while the DB is up**.
|
||||
HealthCmd must contain only space-free tokens
|
||||
(`HealthCmd=pg_isready -d authentik -U authentik`). If health reads
|
||||
unhealthy but logs look fine, compare the baked
|
||||
`{{.Config.Healthcheck.Test}}` array against the intended command before
|
||||
believing the status.
|
||||
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
|
||||
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
## 5. Rootless Quadlet log retrieval
|
||||
|
||||
Step 0 — establish ownership: `systemctl --user status <name>.service`
|
||||
(works ⇒ rootless user unit); `loginctl show-user <user> -p Linger`. If
|
||||
`--user` knows the unit, EVERY command runs as that user with NO sudo.
|
||||
|
||||
| Need | Command |
|
||||
|---|---|
|
||||
| Current instance tail/follow | `podman logs --tail 200 / -f <name>` |
|
||||
| Across restarts/recreates | `journalctl --user -u <name>.service -n 200 --no-pager` |
|
||||
| Time-bounded / crash-only | `--since "14:30"` / `-p warning` |
|
||||
| Kernel-side evidence (OOM, SVM) | `sudo journalctl -k --since …` |
|
||||
|
||||
Quadlet units commonly run `--replace --rm`: **`podman logs` is destroyed on
|
||||
recreation** — the journal is the only durable record. Traps:
|
||||
`sudo podman logs` → "no such container" (root's store does not contain
|
||||
rootless containers; same for `sudo podman ps`); a system-level
|
||||
`journalctl -u <name>` returns nothing for a user unit; during first-start
|
||||
weight downloads health reports unhealthy BY DESIGN (check the payload file
|
||||
is growing). Verbose vendor switches are a config change — through git and
|
||||
deploy.sh, never a live env poke.
|
||||
|
||||
## 6. Compose bring-up with ground truth (rootless hosts)
|
||||
|
||||
1. **Audit reality before pressing buttons:** `podman ps -a` runs per-user —
|
||||
check every namespace (`sudo podman ps -a`) and native units
|
||||
(`systemctl list-units`) before declaring a service absent. Compare
|
||||
compose mount sources against the real filesystem — compose files name
|
||||
phantom paths (e.g. `/mnt/media/*` vs the real LUKS array at
|
||||
`/mnt/entropy`). Fix compose through git first, deploy second.
|
||||
2. **Pre-create bind-mount sources** (`sudo mkdir -p` + `chown 1000:1000`)
|
||||
or creation fails `statfs …: no such file or directory`.
|
||||
3. **Staged bring-up:** infra first (`up -d mariadb redis`), poll
|
||||
`{{.State.Health.Status}}`, then the rest. Fresh containers answer
|
||||
000/refused for the first seconds — poll.
|
||||
4. **Never guess image tags:** ghcr.io answers 403 on the bearer-token
|
||||
request for nonexistent/private repos ("manifest unknown"/token len 0 =
|
||||
repo or tag does not exist, not auth). Query the registry API
|
||||
(`ghcr.io/token?scope=repository:<ns>/<repo>:pull` →
|
||||
`/v2/<ns>/<repo>/tags/list`).
|
||||
5. **Read the interface from inside the image** before guessing flags/envs:
|
||||
`podman exec <ctr> cat /z/cpp.sh` etc. HTTP 500 with a styled error page
|
||||
means RUNNING-but-misconfigured, not down. Copyparty ac ground truth:
|
||||
entrypoint `/bin/ash /z/cpp.sh -c /z/initcfg`; accounts/volumes are ARGV
|
||||
flags (`-a USER:PASS`, `-v SRC:DST:PERMS`), not envs — and with none
|
||||
configured it serves cwd read/write for everyone (an unauthenticated
|
||||
world-writable share).
|
||||
6. Remote-shell quoting: zsh eats bare `===`/`?` tokens; a zero-match from a
|
||||
mangled pattern is not absence — re-verify the target exists.
|
||||
7. `user: "1000:1000"` SIGSEGV at boot = unmapped uid in the rootless userns
|
||||
(host 1000 == container 0): drop the user line, run as container-root.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deployer that installs these units
|
||||
- [[Stale Signal Audit]] — the audit that catches drift's victims
|
||||
- [[Container Network Namespaces]] — reachability between containers
|
||||
- [[Dankclaw/ComfyUI]] — the 9,146-restart incident
|
||||
- [[Infrastructure/Observability]] — same class, twice
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
type: reference
|
||||
title: Stale Signal Audit
|
||||
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
|
||||
tags: [testing, observability, audit, signals, gates]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Stale Signal Audit
|
||||
|
||||
Run after any change and before trusting a green gate. The productive question
|
||||
is **which signal here cannot possibly be correct?** Every pass on this repo
|
||||
(4 of 4) found real defects — including one in the rule file every future
|
||||
agent reads first.
|
||||
|
||||
## The surfaces, and what each vantage point can reach
|
||||
|
||||
| Surface | Producer's vantage point | Can reach |
|
||||
|---|---|---|
|
||||
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
||||
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
|
||||
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
||||
|
||||
Anything else — another host's loopback, a container-internal port never
|
||||
published — is unreachable no matter how healthy the service is. The failure
|
||||
mode is always the same shape: a signal that **cannot** succeed by
|
||||
construction, which teaches the operator to ignore the surface it lives on.
|
||||
|
||||
## 1. Permanently-red probes (reachability)
|
||||
|
||||
For every monitor/probe/alert target ask: *can this checker physically reach
|
||||
that endpoint?* Seen classes: a card probing another host's container-internal
|
||||
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
||||
`/metrics` that is disabled (Gitea 404).
|
||||
|
||||
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
||||
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
|
||||
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
|
||||
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
||||
mark the homepage card `wip: true` with the reason in `description` (the
|
||||
checker skips `wip`; `wip` is the existing convention for "real service,
|
||||
not probeable"), or repoint the job at something reachable. NEVER open a
|
||||
database port on the LAN to make a badge green, and never delete a scrape
|
||||
job — an absent scrape and a healthy one are indistinguishable.
|
||||
- Alert rules must not depend on always-red signals — scope them to services
|
||||
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
|
||||
exactly that reason).
|
||||
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
||||
`connection refused` to `unsupported Content-Type "text/html"` means the
|
||||
service came up and only the scrape is wrong. Read the error, not just
|
||||
up/down.
|
||||
|
||||
## 2. Vacuous assertions (highest severity — the gate is already dead)
|
||||
|
||||
A seam that greps a file passes *by absence* once the file is retired:
|
||||
`grep` exits 2 on a missing path and `!` inverts it.
|
||||
|
||||
```bash
|
||||
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
||||
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
||||
```
|
||||
|
||||
Fix shape — one place asserts the invariant, each local site keeps a live
|
||||
check:
|
||||
|
||||
```bats
|
||||
@test "no <host> compose file competes with the quadlet units" {
|
||||
local cf="$REPO_ROOT/<host>/compose.yaml"
|
||||
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
||||
}
|
||||
# at the per-service site: still fails if the entry returns
|
||||
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
||||
```
|
||||
|
||||
**Prove teeth:** recreate the offending file
|
||||
(`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams,
|
||||
confirm RED, then move the file to trash. A re-armed check you have never
|
||||
seen fail is still unverified.
|
||||
|
||||
## 3. Gates that never run
|
||||
|
||||
Playwright specs in `tests/e2e/` are matched by **number prefix** in
|
||||
`playwright.config.ts`; an unregistered spec silently never runs and the
|
||||
suite count stays the same. Count tests before/after;
|
||||
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
|
||||
|
||||
## 4. Validate the checker before acting on it
|
||||
|
||||
A checker's output is a hypothesis about another tool's semantics. Modelling
|
||||
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
||||
real — the rest were case-sensitivity (a `qBittorrent` link to
|
||||
`qbittorrent.md`), path-qualified resolution (a path-qualified
|
||||
`folder/Note` link resolves by path SUFFIX, not
|
||||
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
|
||||
re-derive the report, only then edit documents. The result lives in
|
||||
`scripts/check-vault-links.py` with those three rules in its docstring,
|
||||
gated by the `no unresolved links in the docs vault` seam. Same class: a
|
||||
`find … | head -1` selector that reads the wrong file; a truncated listing
|
||||
that makes you "fix" a file that exists.
|
||||
|
||||
## 5. Prose invalidated by your own change
|
||||
|
||||
After retiring a file or adding a feature, grep for EVERY reference, not just
|
||||
code:
|
||||
|
||||
```bash
|
||||
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
||||
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
||||
```
|
||||
|
||||
This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
|
||||
after it was restored with GPU access; a unit said "there are no alerting
|
||||
rules" hours after five were added; a docs page documented three scrape jobs
|
||||
as working when they could never report up. **Clean cutover includes prose** —
|
||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||
be stale).
|
||||
|
||||
## Order and verification
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||
gates, docs/rule files, dashboard panels).
|
||||
2. For each, ask what the signal CAN express, not what it says.
|
||||
3. Validate any checker before editing the things it flags.
|
||||
4. Fix at the root; never weaken an assertion to make it pass.
|
||||
5. Gate the class so it cannot return (a seam), then prove the seam fails on
|
||||
the defect.
|
||||
|
||||
Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
|
||||
Prometheus panel expression against the API (0 series on a panel that should
|
||||
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
|
||||
deployed `/srv/homepage/status.json` and triage every `state != up` entry
|
||||
against reality; query `/api/v1/targets?state=active` and print
|
||||
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
|
||||
Report per finding: the surface, why the signal could not have been correct,
|
||||
the fix, and the red/green evidence — flag any regression you introduced
|
||||
yourself in the same breath (one was found this way: a tightened bind killed
|
||||
a reverse-proxied vhost while every gate stayed green, because the spec
|
||||
covering that hostname did not exist).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Testing Policy]] — the seam inventory this audit walks
|
||||
- [[Observability]] — the scrape/probe surfaces it audits
|
||||
- [[Quadlet Lifecycle]] — the ownership labels the probes read
|
||||
Reference in New Issue
Block a user