docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank committed
1 parent
8618ec6dbe
commit
33eae71d33
43 files changed
+5436
-2
No files matched your search
@@ -0,0 +1,128 @@
|
||||
---
|
||||
type: reference
|
||||
title: Container Host Reachability
|
||||
description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
|
||||
tags: [podman, netavark, networking, rootless, reachability, caddy]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Container Host Reachability
|
||||
|
||||
Two failure families on rootless podman + netavark hosts (Dankclaw), both
|
||||
reading as "service down" when the service is up and the *path* is wrong.
|
||||
Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network
|
||||
Namespaces.md` (commit `2e179b0`).
|
||||
|
||||
## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly
|
||||
|
||||
A container cannot reach a `Network=host` service at the host's LAN IP.
|
||||
netavark DNATs **published** ports only: from a bridge container,
|
||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
||||
gateway `10.89.1.1` are both refused. This is what 502'd
|
||||
`hermes.franksdank.cloud` until the Caddyfile upstream became
|
||||
`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the
|
||||
LAN IP was open from that bridge; it never was.
|
||||
|
||||
Probe that proves it in one command:
|
||||
|
||||
```bash
|
||||
ssh <host> "podman run --rm --network <net> --entrypoint sh \
|
||||
docker.io/library/busybox:latest -c \
|
||||
'for p in <target-port> <published-port>; do
|
||||
nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
|
||||
done'"
|
||||
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'
|
||||
```
|
||||
|
||||
Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any
|
||||
failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
|
||||
refusal-vs-timeout matters.
|
||||
|
||||
Fix: point the service's host-facing URL at
|
||||
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
|
||||
git (`deploy.sh <target> --service <svc>`), verify
|
||||
`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
|
||||
**Do not publish the daemon port** to make the LAN IP work — publishing is
|
||||
exactly what makes other ports refuse.
|
||||
|
||||
## Quirk 2 — addressability is per netns, not per host
|
||||
|
||||
- `NetworkMode=host` → the container **is** the host netns: dials all of
|
||||
`127.0.0.1`, binds any free port.
|
||||
- bridge/named net → own netns; only published ports are reachable, via the
|
||||
network's **gateway IP**. Unpublished host listeners refuse **instantly
|
||||
(RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do
|
||||
not hunt for REJECT rules; probe.
|
||||
|
||||
Probe sequence (read-only):
|
||||
|
||||
```bash
|
||||
podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
|
||||
sudo ss -ltnp 'sport = :<port>' # host netns only — why 0.0.0.0 can still be unreachable
|
||||
podman exec <name> cat /proc/net/route # default route = the gateway IP to dial
|
||||
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'
|
||||
```
|
||||
|
||||
Interpretation without blurring:
|
||||
|
||||
- **instant refusal** = nothing listening in a reachable namespace (or
|
||||
rootlessport rejected an unpublished port map).
|
||||
- **timeout** = no route to that address at all. `host.containers.internal`
|
||||
(`169.254.1.2`) is an alias on the **default `podman` net only**; containers
|
||||
on a named network time out against it.
|
||||
|
||||
Tool gotchas (podman 5.x, verified live):
|
||||
|
||||
- `podman network inspect <net> --format '{{json .containers}}'` errors
|
||||
(`can't evaluate field containers in type interface {}`) — list membership
|
||||
from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`.
|
||||
- An **empty** `Networks` field = host network or shared namespace
|
||||
(`--network=container:<other>`), not "no network".
|
||||
- Quadlet `Network=<name>.network` yields the runtime name **`systemd-<name>`**
|
||||
— probe with `podman network ls`, never the `.network` filename.
|
||||
|
||||
Fix options when a bridge-net container must reach a host-netns listener:
|
||||
(a) move it to `Network=host`, or (b) publish the port — (b) makes the port
|
||||
reachable from every namespace **and the LAN**, an exposure change needing
|
||||
explicit operator approval, never a silent fix.
|
||||
|
||||
## Related trap — the pasta port gate (rootless published ports)
|
||||
|
||||
On hosts publishing ports through pasta, the port is bound in **userspace at
|
||||
container CREATE**, not when the app binds. A predecessor stopped moments
|
||||
earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||
then looks fine, and the missing gate stays invisible because the retry masks
|
||||
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
|
||||
Quadlet.
|
||||
|
||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||
replacement is itself the port-holder — without that stop the gate deadlocks
|
||||
against itself), then wait for the port to be empty across **all socket
|
||||
states**:
|
||||
|
||||
```bash
|
||||
wait_port_free() {
|
||||
local port="$1" limit="${2:-90}" i
|
||||
for ((i = 1; i <= limit; i++)); do
|
||||
if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "FAIL: port $port still held after ${limit}s:" >&2
|
||||
ss -tan "sport = :$port" >&2 || true
|
||||
return 1
|
||||
}
|
||||
```
|
||||
|
||||
`sport` catches accepted-connection server sockets and TIME_WAIT residue that
|
||||
`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate
|
||||
by re-running the same deploy against a running service (idempotency) and
|
||||
assert it in seam tests (`wait_port_free` present in the deploy handler).
|
||||
|
||||
## Related
|
||||
|
||||
- [[Quadlet Lifecycle]] — the unit layer these containers run under
|
||||
- [[Container Network Namespaces]] — fleet network namespace source of truth
|
||||
- [[Observability]] — the alerting paths that depend on reachability
|
||||
- [[Deploy State Triage]] — the deploy flow that installs these configs
|
||||
@@ -0,0 +1,331 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Audit and Posture
|
||||
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
|
||||
tags: [audit, security, prometheus, caddy, monitoring, git]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Audit and Posture
|
||||
|
||||
## Monitoring gap audit
|
||||
|
||||
Use after any reboot, or whenever Prometheus targets look wrong. Two failure
|
||||
modes, only the first announces itself.
|
||||
|
||||
**1. A service that should exist and doesn't is invisible.** A missing
|
||||
exporter shows up as nothing — no failing target if the job was configured
|
||||
for a host that no longer serves, no alert because the series is absent. So
|
||||
check **existence, not health**:
|
||||
|
||||
```bash
|
||||
podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones
|
||||
ls ~/.config/containers/systemd/*.container # what should be boot-persistent
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
|
||||
case: rootless podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and
|
||||
stays dead. If the compose file still names it, the repo's own rule is being
|
||||
violated — fix by writing the quadlet (mirror the same service on the other
|
||||
host), deleting the compose entry (one owner per container name), and making
|
||||
the deploy path reach it: planner service-set + path rule, dispatcher
|
||||
allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
|
||||
a hardcoded peer's path.
|
||||
|
||||
**2. Classify every down target — do not report a count.**
|
||||
`targets?state=active` gives health **and** last error; two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
|
||||
| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
|
||||
| target absent | §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics # hand-probe before believing the scraper
|
||||
```
|
||||
|
||||
"7 of 11 up" without the four reasons is how a real outage hid behind three
|
||||
config defects on this fleet — the crash-looping GPU service was one of four
|
||||
failures reported as a number.
|
||||
|
||||
**3. Exposure posture while you are in there.** An exporter bound to the
|
||||
wildcard publishes an unauthenticated API on every interface. Expect exactly
|
||||
two listeners per host-exporter — loopback for host-local probes, the host's
|
||||
own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
|
||||
show two explicit addresses, never `0.0.0.0`.
|
||||
|
||||
**4. Leave decisions that are not yours.** Fix mechanical drift (missing
|
||||
unit, wrong path, wildcard bind). Do NOT disable a probe to green a
|
||||
dashboard, or "fix" a crash-looping service whose posture depends on owner
|
||||
intent — capture the evidence (restart count, exit code, last error line) and
|
||||
hand it over intact.
|
||||
|
||||
## vhost proxy health probes
|
||||
|
||||
After deploying or auditing a Caddy/Anubis front door, verify **from a
|
||||
client**, not from the backend's localhost — that is where transport
|
||||
mismatches, auth walls, and dead upstreams show up.
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
|
||||
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
|
||||
```
|
||||
|
||||
A status code is a diagnosis, not a failure:
|
||||
|
||||
- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
|
||||
bypasses auth by design) — probe `/` to decide.
|
||||
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
|
||||
there = healthy).
|
||||
- `502` = Caddy could not reach/negotiate with the upstream: dead backend
|
||||
**or transport mismatch**. Probe the backend port directly from the proxy's
|
||||
network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
|
||||
vhost 502 ⇒ config bug, not an outage.
|
||||
- Login page to anonymous (grep body for `password`) = auth ON; instant
|
||||
redirect to `/transferlist/`-style routes = auth OFF = open door.
|
||||
|
||||
**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
|
||||
Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
|
||||
→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
|
||||
drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
|
||||
HTTP off-loopback and needs a TLS transport. Match the transport to what the
|
||||
backend speaks.
|
||||
|
||||
**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
|
||||
**1 when zero mismatches** — inside an `&&` chain it silently kills
|
||||
everything after it while still printing `0`. Wrap:
|
||||
`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.
|
||||
|
||||
## Security posture audit (behavioral)
|
||||
|
||||
Answers "what can the internet touch, and is everything else actually
|
||||
gated?" — **all findings must be behavioral probes**, not config reads.
|
||||
|
||||
**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
|
||||
vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
|
||||
redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
|
||||
@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
|
||||
DNS absence.
|
||||
|
||||
**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
|
||||
`curl -sk https://<vhost>/` and grep the body for
|
||||
`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
|
||||
with NO markers means the app is served directly — the guard is bypassed.
|
||||
|
||||
**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
|
||||
401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
|
||||
→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
|
||||
→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
|
||||
`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
|
||||
`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
|
||||
`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
|
||||
`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
|
||||
→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
|
||||
flow URL is the proof).
|
||||
|
||||
**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
|
||||
unreliable; scan from gluetun's ProtonVPN egress:
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
|
||||
8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
|
||||
5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
|
||||
>/dev/null 2>&1 && echo OPEN: \$p; done"'
|
||||
```
|
||||
|
||||
Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
|
||||
→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
|
||||
(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
|
||||
with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
|
||||
errors instantly and the `&&` chain silently skips the echo (a sub-second
|
||||
wall time on a "timeout" scan means the flag failed, not that ports closed).
|
||||
|
||||
**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
|
||||
http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
|
||||
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
|
||||
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
|
||||
auth (OpenAI API, no token). Stale-record check: current WAN IP =
|
||||
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
|
||||
compare against every public A record — a record at a different Comcast IP
|
||||
is stale.
|
||||
|
||||
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
|
||||
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
|
||||
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
|
||||
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
|
||||
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
|
||||
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
|
||||
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
|
||||
(dead config, no exposure).
|
||||
|
||||
## Stash supersession audit before drop
|
||||
|
||||
Use when a repo (especially a live-deployment checkout where the working tree
|
||||
IS the deployment) has an old stash whose content may have been superseded by
|
||||
later commits, and dropping blindly risks losing unique facts.
|
||||
|
||||
1. **Inventory**: `git stash show --stat stash@{0}`.
|
||||
2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
|
||||
diff against HEAD; count lines present only in the stash version. A file
|
||||
with zero unique-vs-HEAD lines is fully superseded — safe to drop.
|
||||
3. **Classify each unique-content hit**: *stale fact* (superseded by a later
|
||||
refactor, e.g. old model naming) → drop is correct; *live durable fact*
|
||||
(an operational invariant never re-recorded) → extract it into live
|
||||
docs/memory or flag to the user before dropping.
|
||||
4. **Archive before dropping** — zero loss: tag the stash commit, then drop:
|
||||
|
||||
```bash
|
||||
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
|
||||
git stash drop "stash@{0}"
|
||||
```
|
||||
|
||||
The tag pins the full stash object including its index parent; recoverable
|
||||
via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
|
||||
for the untracked-files third parent. Stashes are invisible in normal history
|
||||
and expire from reflogs (~90 days); tagging converts "drop" into a
|
||||
reversible archive at zero workflow cost.
|
||||
|
||||
## Whole-fleet audit (git/test/live/docs cross-check)
|
||||
|
||||
Deep-audit the homelab fleet — git/test state, live service health, and
|
||||
docs-vs-live cross-check. Run from the dev checkout
|
||||
(`/home/blank/Projects/serverAppConfig`).
|
||||
|
||||
**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
|
||||
trust the display order of parallel SSH outputs: they can render crossed and
|
||||
you will attribute host A's output to host B (2026-09-21: a false host↔IP
|
||||
"conflict" dissolved by a single self-contained per-host probe). Put
|
||||
`hostname` + a distinctive marker + the LLM identity in the SAME command:
|
||||
|
||||
```
|
||||
for ip in 192.168.0.143 192.168.0.235; do
|
||||
ssh -o BatchMode=yes dank@$ip \
|
||||
'echo "host=$(hostname)"; \
|
||||
echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
|
||||
echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
|
||||
done
|
||||
```
|
||||
|
||||
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
|
||||
= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
|
||||
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
|
||||
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
|
||||
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
|
||||
it is unaffected by IP drift.
|
||||
|
||||
**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
|
||||
`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
|
||||
the pass condition.
|
||||
|
||||
**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
|
||||
end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
|
||||
fleet. **A single-test flake is normal**, not a regression: the plex crawl
|
||||
intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
|
||||
times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
|
||||
that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
|
||||
to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
|
||||
while the change itself is fine — verify the actual change live, then re-run
|
||||
the suite.
|
||||
|
||||
**4. Repo state.** `git status --short --branch; git log --oneline -20;
|
||||
git stash list; git worktree list` (prune stale worktrees), and
|
||||
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
|
||||
**shared checkout with parallel instances**: uncommitted peer WIP and
|
||||
stash-list churn are expected. Never commit or drop a peer's files. To
|
||||
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
|
||||
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
|
||||
|
||||
**5. Prometheus — expected-down targets (not outages).** Query
|
||||
`/api/v1?query=up` from inside the container; the structurally red-by-
|
||||
construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
|
||||
`gitea` exporter off) are documented in [[Observability]] — anything else
|
||||
down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
|
||||
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
|
||||
a *host bind*, not a published port, so the bridge-networked Prometheus
|
||||
container cannot reach it at the host's LAN IP — netavark DNATs published
|
||||
ports only. Scrape same-host services via
|
||||
`host.containers.internal:<port>`; verify the fix from inside the container
|
||||
before editing: `podman exec prometheus wget -qO-
|
||||
http://host.containers.internal:<port>/metrics`.
|
||||
|
||||
**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
|
||||
stack mapping against step 1. The LLM endpoint table and `models.yml` are
|
||||
probed live and stay current; the prose host labels are the part that goes
|
||||
stale after an IP move. Report discrepancies; do not "fix" prose from a
|
||||
single crossed probe.
|
||||
|
||||
## Bulk-redact an identity from a docs + config tree
|
||||
|
||||
Removing a retired host / machine / project name from a repo that holds both
|
||||
documentation and live config. Evidence-first, surgical, reversible.
|
||||
|
||||
**1. Inventory before touching anything.** Count hits **per file** (the
|
||||
count decides whether this is a 5-minute job or a user-input decision),
|
||||
excluding logs/`*.db`/runtime state, and capture `git status --short` first
|
||||
— you must know which files carry in-flight peer WIP so you never stage
|
||||
them.
|
||||
|
||||
**2. Classify every hit:**
|
||||
|
||||
| Bucket | Test | Action |
|
||||
|---|---|---|
|
||||
| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
|
||||
| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
|
||||
| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |
|
||||
|
||||
Load-bearing examples that look like prose but are not: `install.sh` doing
|
||||
`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
|
||||
*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
|
||||
model strings, driver versions, PCI addresses are subject matter, not
|
||||
identity. State the boundary in the commit message.
|
||||
|
||||
**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
|
||||
tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
|
||||
genuinely different. When distilling: read every page, sort into
|
||||
still-true-and-machine-independent (fold into a general page) vs
|
||||
specific-to-the-retired-box (let go), then `mv` the directory to
|
||||
`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
|
||||
filename anywhere in the tree, so deleting a page can break a link in a
|
||||
file you never grepped — run the link checker.
|
||||
|
||||
**4. Execute with count assertions, never blind sed.** Per-file mapping of
|
||||
`(path, old_exact_string, new_string, expected_count)`; abort the entry if
|
||||
the count differs; exact strings, not regex. **Do not hand this to a
|
||||
subagent** — observed: dropped list markers so bullets stopped rendering, a
|
||||
numbered list with two `3.` items, "corrected" paths to nonexistent
|
||||
directories, a deleted sentence where a reword was required. If a subagent
|
||||
already did it: `git checkout --` its touched paths (excluding the WIP
|
||||
files from step 1) and redo it.
|
||||
|
||||
**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
|
||||
`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
|
||||
— the diff is the step that catches damage; the grep only proves a string
|
||||
is gone. Then the repo's own linter.
|
||||
|
||||
**6. Commit scoped** — only your own paths, never `git add -A`. If the
|
||||
checkout IS the live deployment, committing on the current branch is usually
|
||||
correct; a branch switch can swap live config out from under a running
|
||||
process. Push/PR only when asked.
|
||||
|
||||
**7. Report the residue honestly** — what still carries the name and why:
|
||||
excluded scopes, load-bearing config, git history, and the cost of rewriting
|
||||
history (force-push + re-clones). "Purged from the working tree" is a true
|
||||
claim; "purged" alone is not.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — the monitored stack
|
||||
- [[Stale Signal Audit]] — the down-target decision table
|
||||
- [[Observability Provisioning]] — adding scrape targets and rules
|
||||
- [[Homepage Status Probes]] — the status page that renders probe states
|
||||
- [[Deploy State Triage]] — the deploy plumbing the audits exercise
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
type: reference
|
||||
title: Fleet Ops Playbook
|
||||
description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host.
|
||||
tags: [operations, incidents, gpu, systemd, wol, forensics]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Fleet Ops Playbook
|
||||
|
||||
## Endpoint down, connection refused — GPU holder reclaim
|
||||
|
||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
||||
`frank-small` (dankclaw, root unit), `frank-large` (dankclaw, rootless
|
||||
Quadlet).
|
||||
|
||||
**1. Classify the stop before touching anything**
|
||||
|
||||
```bash
|
||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||
```
|
||||
|
||||
`inactive` + `enabled` + unmasked + `NRestarts=0` → **intentional stop**,
|
||||
never restored. No crash; do not look for segfaults. `NRestarts>0` /
|
||||
`activating` → different path; read the journal first.
|
||||
|
||||
**2. Are the GPUs actually free?** A check, never an assumption:
|
||||
|
||||
```bash
|
||||
rocm-smi --showmeminfo vram | grep Used # idle baseline ~60 MiB/card
|
||||
ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
||||
```
|
||||
|
||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||
prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit (frank-medium 35:30
|
||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
||||
restart on allocation, not config).
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
|
||||
```bash
|
||||
ps -o pid,ppid,user,lstart,etime,comm -p <pid>
|
||||
sudo readlink /proc/<pid>/cwd /proc/<pid>/exe
|
||||
tr '\0' ' ' < /proc/<pid>/cmdline # port, ctx, split
|
||||
sudo ls -l /proc/<pid>/fd # where stdout/stderr go
|
||||
sudo ss -tnp | grep <port>
|
||||
awk '{print $14+$15}' /proc/<pid>/stat; sleep 10; awk '{print $14+$15}' /proc/<pid>/stat
|
||||
```
|
||||
|
||||
Abandoned **only if all** hold: utime+stime **identical** across the 10 s
|
||||
sample (frozen CPU, not just low); **zero** established connections; bind
|
||||
scope `127.0.0.1` (loopback scratch, invisible to endpoint monitors); log
|
||||
file (fd 1/2 target) last written long ago, last lines showing success not a
|
||||
hang. Active CPU or live connections → someone's running session. Stop and
|
||||
ask.
|
||||
|
||||
**4. Reclaim**
|
||||
|
||||
```bash
|
||||
cp -a <logfile> <logfile>.preserved-YYYYMMDD # evidence outlives the kill
|
||||
kill <pid> # SIGTERM first; exits <1 s
|
||||
rocm-smi --showmeminfo vram | grep Used # expect ~60 MiB baseline
|
||||
```
|
||||
|
||||
Never `pkill -f` a pattern present in your own command line.
|
||||
|
||||
**5. Restore and verify — health 200 alone is NOT proof**
|
||||
|
||||
```bash
|
||||
sudo systemctl start <svc>
|
||||
for i in $(seq 1 60); do curl -s -m 5 -o /dev/null -w '%{http_code}\n' \
|
||||
http://127.0.0.1:<port>/health; sleep 10; done # 503 while loading -> 200
|
||||
sudo tr '\0' ' ' < /proc/<MainPID>/cmdline | grep -oE '\-\-(tensor-split|split-mode|ctx-size|port|spec-type|parallel|flash-attn)'
|
||||
```
|
||||
|
||||
Prove with a **real completion** from off-host — content, reasoning, spec
|
||||
counters:
|
||||
|
||||
```bash
|
||||
curl -sS -m 180 http://<host>:<port>/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"<alias>","messages":[{"role":"user","content":"Reply with exactly: OK"}],"max_tokens":64,"temperature":0}'
|
||||
# choices[0].message.content / .reasoning_content / timings.draft_n, draft_n_accepted
|
||||
```
|
||||
|
||||
Record the runtime fingerprint (`<bin> --version` → `b<build>, commit <sha>`)
|
||||
and confirm it is the pinned runtime. Append a dated entry to
|
||||
`serverAppConfigs-docs/log.md`, run `./tests/run.sh`, commit, push (origin is
|
||||
the local Gitea on Dankclaw — push allowed).
|
||||
|
||||
**Gotchas seen here:** dankserver's login shell is ZSH — never start an
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. First request after a halogen (`frank-large`) restart can go silent
|
||||
45–60 s and self-recover; do not restart over it. Rootless `frank-large`:
|
||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||
|
||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||
|
||||
Any scratch/experimental vLLM container on a host that also runs a production
|
||||
LLM service (`frank-primary`, `frank-large`, `frank-small`) MUST stop the
|
||||
production unit first, and the window script must verify it. Two independent
|
||||
failure modes if skipped:
|
||||
|
||||
1. **Expert-offload planner dies on host RAM.**
|
||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||
plan time. A production rank worker holds tens of GiB of unevictable
|
||||
pinned expert rows (frank-primary: ~76 GiB × 2 ranks), so a beside-prod
|
||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
||||
gun.
|
||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||
only once the prod unit is down.
|
||||
|
||||
Window skeleton (dankserver):
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-primary.service
|
||||
sudo systemctl mask --runtime frank-primary.service # relinked unit: plain mask refuses; runtime mask self-clears at reboot
|
||||
sleep 3
|
||||
podman ps --format "{{.Names}}" | grep -q frank-primary-glm && { echo "prod still running - abort"; exit 1; }
|
||||
# ... launch scratch container, watch logs ...
|
||||
podman rm -f <scratch> # stuck vLLM ignores SIGTERM
|
||||
sudo systemctl unmask --runtime frank-primary.service
|
||||
sudo systemctl reset-failed frank-primary.service
|
||||
sudo systemctl start frank-primary.service
|
||||
# restore oracle is a CONJUNCTION: health 200 AND is-active active AND is-enabled enabled AND no /run/systemd/system mask symlink
|
||||
```
|
||||
|
||||
The deploy restores the unit after bench windows, so stop is sufficient
|
||||
there; `mask --runtime` is for scratch boots that must survive a stray
|
||||
restart. Always end a window by re-checking the conjunction before
|
||||
reporting — if a restore after failure doesn't happen automatically, prod
|
||||
stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## Iterating on a fleet service before commit
|
||||
|
||||
The repo's contract is git → `deploy.sh` → host, but live testing needs the
|
||||
new file on the host first, and the naive way (copy into the host checkout)
|
||||
blocks the very deploy that would install it. The loop:
|
||||
|
||||
1. Edit locally; run `./tests/run.sh`.
|
||||
2. Ship working files **for testing**:
|
||||
`cat local/file | ssh <host> 'cat > /home/dank/serverAppConfigs/<path>'`
|
||||
(then `chmod +x` if executed — `cat >`/scp drop the exec bit and a
|
||||
container entrypoint fails status 126, "found but not executable").
|
||||
3. Test against the live service; iterate.
|
||||
4. **Before committing**, prove the host copy is byte-identical to what you
|
||||
are about to commit (`sha256sum` both ends) — otherwise step 6 loses work.
|
||||
5. Commit and push.
|
||||
6. Clean the host checkout so the deploy can pull: untracked hand copies →
|
||||
`mv` to `/tmp/handcopy/`; modified tracked files → `git checkout -- <paths>`
|
||||
(safe only because of step 4); confirm `git status --short` prints nothing.
|
||||
7. `./scripts/deploy.sh <target> --service <svc>`.
|
||||
|
||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||
working tree files would be overwritten by merge" — exactly the file you
|
||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||
container restart with no deploy; root-owned files from a container write
|
||||
need `chown` back before git operations; and **a restart is not a
|
||||
verification** — verify the *specific* claim (new code path's own output,
|
||||
file checksum on the host, the API serving the resource).
|
||||
|
||||
Why not keep a scratch copy on the host? The deployed artifact must be the
|
||||
committed one. Probes may live anywhere; runtime state lives in git.
|
||||
|
||||
## Incident window reconstruction
|
||||
|
||||
Use when the question is **"what was running when it broke?"** and the answer
|
||||
must come from artifacts, not recollection. Three independent clocks,
|
||||
cross-checked:
|
||||
|
||||
1. **Per-run duration** — scripts wrap commands in `time` → `real 4m19.129s`.
|
||||
2. **Log mtime** — last write of the log.
|
||||
3. **Journal first-failure instant** — first error line from the affected
|
||||
service.
|
||||
|
||||
`start ≈ mtime − real` for the last run in a log; each run nests inside the
|
||||
previous one's window; the run whose `[start, end]` contains the journal's
|
||||
first-failure instant is the answer.
|
||||
|
||||
```bash
|
||||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||||
ssh <host> 'grep -aE "^####|^\+ /path/to/binary" <logdir>/<name>.log | sed "s/\(.\{130\}\)/\1\n /g"'
|
||||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||||
```
|
||||
|
||||
Traps (all hit for real): `/tmp` does not survive a reboot — launch
|
||||
experiment scripts from a persistent path (`/home/dank/models/logs/<tag>.log`
|
||||
survived while `/tmp/x4*.sh` died); a log's mtime is not proof a run finished
|
||||
(truncation with the fd still held keeps growing); long command lines are
|
||||
display-truncated — wrap before reading or you report parameters never used;
|
||||
off-by-one filenames (`x4b.log` vs `x4b-experiment.log`) silently skip
|
||||
evidence — list the directory and match exactly; **overlap is a hypothesis,
|
||||
not an observation** unless a sampler proves it; do not invent the cause —
|
||||
reconstruct the *window* and keep the *mechanism* separate (`journalctl -k |
|
||||
grep -i oom`); publish "trigger not established" when the proof is absent.
|
||||
|
||||
Fleet-specific: container healthchecks shell out to the **podman CLI**, so
|
||||
the first `invalid internal status … could not find any running process` in
|
||||
the user journal is an early, per-minute indicator of rootless-podman state
|
||||
death — it timestamps the incident to within one healthcheck interval (how
|
||||
the 18:58:30 Dankclaw break was pinned to a bench still loading).
|
||||
|
||||
Worked example (2026-09-16, Dankclaw): log mtimes x4e-run 18:58:29 /
|
||||
x4f-run 18:59:56; durations E1 `real 4m53.946s`, F `real 5m24.785s` (exit 143
|
||||
= killed); journal first failure 18:58:30 ⇒ **F** was in flight (launched
|
||||
18:54:31, inside its window).
|
||||
|
||||
## Wake-on-LAN for a sleeping fleet host
|
||||
|
||||
A fleet host is asleep (100% ping loss; `ip neigh show <ip>` shows STALE or
|
||||
FAILED) and you need it up — typically the FrankBook-Pro Mac mini
|
||||
(192.168.0.112, **MAC `de:6b:6d:c1:52:23`**, sleeps and breaks the e2e
|
||||
`11-admin` spec).
|
||||
|
||||
- **Find the MAC**: `ip neigh show <ip>` — even a STALE entry carries it;
|
||||
fallback Pi-hole/DHCP leases or fleet docs.
|
||||
- **Send the packet**: `6×0xFF + 16×MAC` = 102 bytes, UDP to port **9** (and
|
||||
7), broadcast. The dev workstation has **two interfaces on 192.168.0.0/24**
|
||||
(Wi-Fi + wired USB-Ethernet, default route metric 100) — a plain
|
||||
`sendto("255.255.255.255", 9)` leaves only the default-route interface.
|
||||
Bind explicitly to each interface's local IP and target the subnet
|
||||
broadcast `192.168.0.255` as well as `255.255.255.255`; burst ~15 packets
|
||||
per interface ~0.2 s apart. `wakeonlan` is not installed; use a raw UDP
|
||||
socket with `SO_BROADCAST=1`:
|
||||
|
||||
```python
|
||||
import socket, time
|
||||
MAC = "de:6b:6d:c1:52:23"
|
||||
mac = bytes(int(x,16) for x in MAC.split(":"))
|
||||
pkt = b"\xff"*6 + mac*16 # 102 bytes
|
||||
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
|
||||
("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
|
||||
s.bind((local, 0))
|
||||
for _ in range(15):
|
||||
s.sendto(pkt,(dest,9)); s.sendto(pkt,(dest,7)); time.sleep(0.2)
|
||||
s.close()
|
||||
```
|
||||
|
||||
- **Poll for the wake**: a Mac mini takes **30–90 s**; `ping -c 1 -W 1` every
|
||||
5 s for ~2–3 min before concluding failure.
|
||||
- **If it does not wake** (the common case): macOS honors WoL only when BOTH
|
||||
are enabled on the mini — **System Settings → Battery → Options → "Wake
|
||||
for network access"** = ON, and the active port's **"Enable Wake for LAN"**
|
||||
(Network → [interface] → Details). If either is off, no packet from any
|
||||
source wakes it, and you cannot change it while asleep. A full power-off
|
||||
(smart plug cycled) ignores WoL entirely. Ask the user to toggle the
|
||||
setting, then re-fire.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
|
||||
- [[Remote Job Runner]] — supervising the long jobs that create these windows
|
||||
- [[Container Host Reachability]] — the pasta port gate at container create
|
||||
- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
|
||||
- [[Model Swap and Rollback]] — swap-time incidents and rollbacks
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
type: reference
|
||||
title: Payload and Disk Reclaim
|
||||
description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification.
|
||||
tags: [operations, disk, podman, payloads, huggingface, trash]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Payload and Disk Reclaim
|
||||
|
||||
Standing procedure for "clean up old models" / "disk is full" on a
|
||||
serverAppConfigs host. Order matters: **measure, prove nothing live uses it,
|
||||
stage, ask.** Staging frees nothing until the user approves the purge — say so
|
||||
up front or the request reads as unfulfilled.
|
||||
|
||||
## 1. Measure the whole filesystem, not the category asked about
|
||||
|
||||
```sh
|
||||
du -sh /home/dank/models /home/dank/.local /home/dank/.cache \
|
||||
/home/dank/comfyui-data /home/dank/GitPackages 2>/dev/null
|
||||
df -h /home/dank | tail -1
|
||||
podman system df # reclaimable image space
|
||||
du -sh ~/.local/share/containers # where rootless storage physically is
|
||||
```
|
||||
|
||||
The answer is usually NOT the category the user named. On Dankclaw
|
||||
(2026-09-28) retired models were 510G, but podman images were **279G
|
||||
reclaimable (95%)** and uv+pip caches **73G** — 352G of regenerable garbage,
|
||||
bigger than the models. Lead with that finding; it changes the decision.
|
||||
|
||||
Check for sparse files before claiming a saving:
|
||||
|
||||
```sh
|
||||
stat -c 'apparent=%s blocks=%b' <file>; du -sh --apparent-size <file>; du -sh <file>
|
||||
```
|
||||
|
||||
A 3.3G "log" (`logs/x4e-run.log`) was **4K real** (8 blocks × 512) — deleting
|
||||
it frees nothing. `du` vs `du --apparent-size` finds these instantly.
|
||||
|
||||
## 2. Classify against the runbook index
|
||||
|
||||
Every payload directory maps to a runtime in the AGENTS.md retired-runbook
|
||||
table. Three buckets:
|
||||
|
||||
- **live** — referenced by a running service's env/unit.
|
||||
- **rollback** — named as a rollback target in a runbook. ASK before
|
||||
touching: they are the difference between a one-minute flip and a 30G
|
||||
re-download.
|
||||
- **retired** — maps to a retired runtime. Safe candidates.
|
||||
|
||||
Trap: a directory can hold **both** a dead target and a live drafter
|
||||
(frank-large's DFlash2 drafter lives beside the previous target). Move the
|
||||
single file, never the directory, and say so in the manifest.
|
||||
|
||||
## 3. The live-reference check — before moving anything
|
||||
|
||||
```sh
|
||||
for d in <candidate dirs>; do
|
||||
grep -rl "$d" --include="*.sh" --include="*.env" --include="*.service" \
|
||||
--include="*.container" --include="*.yaml" . | grep -v '^./old/'
|
||||
done
|
||||
```
|
||||
|
||||
Then **read every hit**; do not count them. Most are benign (comments,
|
||||
retired fetch-script destinations, `retire_*_quadlet` helpers that move a
|
||||
quadlet not a model) — but on Dankclaw this check found a live bug:
|
||||
`serve-frank-small.sh`'s `: "${MODEL_DIR:=...}"` fallback still pointed at a
|
||||
payload retired a week earlier. The env file always overrode it, so it was
|
||||
invisible; a missing env file would have produced a "missing asset" naming a
|
||||
model that no longer exists.
|
||||
|
||||
**Durable lesson:** `:`-defaults in a serve script are never exercised in
|
||||
production, so they rot silently. When reclaiming a payload, grep serve
|
||||
scripts too — the fallback is a live reference in all but name.
|
||||
|
||||
Also check service **env files** (`scripts/*.env`) for `MODEL_DIR`/`MODEL`
|
||||
paths, not just unit `Volume=` lines: a payload can be mounted by path and
|
||||
named by env. `models/logs` and small tool dirs are not cache — logs are
|
||||
evidence, keep them.
|
||||
|
||||
## 4. Stage, never delete
|
||||
|
||||
```sh
|
||||
STAGE=/home/dank/.AGENTTRASH/$(date +%Y%m%d)-<topic>
|
||||
mkdir -p "$STAGE"
|
||||
mv <path> "$STAGE"/
|
||||
```
|
||||
|
||||
Same filesystem ⇒ instant and recoverable — **and reclaims no space**; say so
|
||||
rather than implying space was freed. Write `MANIFEST.md` into the stage dir:
|
||||
per item — path, size, owning runtime, when retired, the `mv` command to
|
||||
restore. Record what you **deliberately kept** and why; that is what a future
|
||||
reader needs. Always check for pre-existing staged trash
|
||||
(`du -sh ~/.AGENTTRASH/*`): 118G from a 2026-09-21 cleanup sat un-purged;
|
||||
report the running total, not just the new batch. Root-owned leftovers need
|
||||
`sudo`, and check `swapon --show` first (a trashed swapfile must not be
|
||||
active).
|
||||
|
||||
## 5. Verify, then report — and do not purge
|
||||
|
||||
```sh
|
||||
systemctl is-active frank-large frank-small
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
Report: staged items and sizes, what was kept and why, what is **still not
|
||||
freed**, remaining candidates untouched. Purge only on explicit approval.
|
||||
|
||||
## Repo-flow gotcha: do NOT push before you deploy
|
||||
|
||||
`scripts/deploy.sh` sets `BASE_REVISION=$(git rev-parse origin/main)`. If you
|
||||
pushed separately, the diff is empty and the planner reports "changed
|
||||
services: (none)" — looks like a planner bug, is not one. The tool is
|
||||
designed for the **unpushed** state: edit → test → `deploy.sh` (it pushes,
|
||||
then dispatches). If you already pushed, pull on the host instead, and reason
|
||||
about whether a restart is actually owed. Any serve-script/env change needs
|
||||
`./tests/run.sh` before commit; a `:`-default change is usually **inert**
|
||||
(env overrides it) and owes no restart — but say so explicitly.
|
||||
|
||||
# Purge and refresh protocol
|
||||
|
||||
When the task is clearing/reclaiming/**refreshing** model caches on a serving
|
||||
host, add these steps.
|
||||
|
||||
## Fingerprint before deleting
|
||||
|
||||
`ls -l --block-size=1 <dir>`; for LLM payloads compare each file against the
|
||||
Hub's declared size. Fingerprints are the only way to answer "did the refresh
|
||||
change anything" afterwards — **a refresh is also an update check**: if
|
||||
upstream never changed, the bytes come back identical, which is itself the
|
||||
finding. Sidecars are versioned **in place**: equal main-checkpoint sizes do
|
||||
not prove the repo is unchanged, and a stale sidecar is invisible from its
|
||||
name (observed: `overlay.hgn` 2,477,677,120 → 2,572,466,560 while the main
|
||||
checkpoint returned byte-identical). If a file is irreplaceable upstream
|
||||
(delisted revision, gated repo), say so before deleting.
|
||||
|
||||
## Confirm re-fetchability first
|
||||
|
||||
Query the Hub anonymously; a gated repo 403s even with the host's token:
|
||||
|
||||
```bash
|
||||
curl -s "https://huggingface.co/api/models/<repo>/tree/main?recursive=true&blobs=true" \
|
||||
| python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d),'entries'); [print(e.get('size'),e['path']) for e in d[:5]]"
|
||||
```
|
||||
|
||||
An error/empty result means it is NOT safe to delete. Filter by **quant
|
||||
name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
||||
`.partNofM` fragments that filter hides.
|
||||
|
||||
## Refresh per engine — know the recovery path BEFORE deleting
|
||||
|
||||
- **frank-large (halogen)**: the engine fetches its own payload via
|
||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
||||
podman refuses a mount whose source is missing
|
||||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
|
||||
at 130–240 MB/s ≈ 10–20 min, then load.
|
||||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||||
without weights. Pull yourself and verify:
|
||||
|
||||
```bash
|
||||
curl -fL -C - --retry 10 -o "$DEST/<file>" \
|
||||
"https://huggingface.co/<repo>/resolve/main/<file>" # -f is mandatory
|
||||
sha256sum "$DEST"/*; head -c 4 "$DEST"/*.gguf # compare to LFS oid + GGUF magic
|
||||
```
|
||||
|
||||
`curl` exits 0 on HTTP 401/403 and writes the error body as the artifact — a
|
||||
gated repo once produced three 153-byte shards with identical sha256s and a
|
||||
cheerful DONE line for 104 GiB. Always `-fL -C -`, compare byte counts to the
|
||||
Hub's declared sizes, compare sha256 to the LFS oid. Assemble `.partNofM`
|
||||
fragments with `cat part1 part2 part3 > name.gguf` in numeric order (per-part
|
||||
hashes verify the whole — concatenation is deterministic). Public repos need
|
||||
no token: omit `-H "Authorization: Bearer …"` from argv entirely (it is
|
||||
world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
|
||||
`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
|
||||
fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
|
||||
every restored endpoint. Read the engine's memory ledger (kv pool / weights /
|
||||
blocks left) rather than assuming the config still fits. Sweep the vhosts too
|
||||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
|
||||
`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
|
||||
whole host shares.
|
||||
|
||||
## Things that bite
|
||||
|
||||
- **Do not run a 100 GB+ re-fetch concurrently with a big iGPU benchmark** —
|
||||
the A* run commits ~116 GiB and leaves ~10 GiB available; cache churn risks
|
||||
the OOM/state failures this fleet has already seen.
|
||||
- **Reboot is the repair of last resort** if rootless podman's pause process
|
||||
dies mid-work (`invalid internal status … could not find any running
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- A same-filesystem trash reclaims no space; root-owned leftovers need
|
||||
`sudo`; never `rm` without explicit permission.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the runbook index this classification reads
|
||||
- [[Quadlet Lifecycle]] — restoring the unit after a payload refresh
|
||||
- [[Remote Job Runner]] — supervising long re-fetches
|
||||
- [[Deploy State Triage]] — the push-before-deploy trap in context
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
type: reference
|
||||
title: Remote Job Runner
|
||||
description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline.
|
||||
tags: [operations, ssh, jobs, sentinel, detached]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Remote Job Runner
|
||||
|
||||
Use for any long-running job on a remote host (benchmarks, model loads,
|
||||
builds, sweeps) whose result must survive the session. Also whenever you catch
|
||||
yourself writing `ssh host 'tail -f log | sed …'` as a watcher.
|
||||
|
||||
## Why the obvious approach fails structurally
|
||||
|
||||
`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways
|
||||
that all LOOK like "the job died":
|
||||
|
||||
- the session broker **aborts long-lived ssh** — the watcher vanishes and the
|
||||
result is lost;
|
||||
- a `ready` regex can match the log's own **header** on the first line — the
|
||||
supervisor reports success instantly while the job is still running;
|
||||
- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe;
|
||||
- output goes into the ssh stream, so an aborted session loses everything.
|
||||
|
||||
## The mechanism: detach on the HOST, poll a sentinel file
|
||||
|
||||
Completion is a **`.done` sentinel touched last**, not a log pattern. Status
|
||||
is then one fast independent command with no long-lived process to lose.
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# job.sh {start|status|wait|out|tail|watch} ... Jobs live in $JOBS.
|
||||
set -uo pipefail
|
||||
JOBS=/home/dank/jobs
|
||||
mkdir -p "$JOBS"
|
||||
|
||||
case "${1:-}" in
|
||||
start)
|
||||
NAME="$2"; shift 2
|
||||
[ "${1:-}" = "--" ] && shift
|
||||
rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
|
||||
# setsid detaches from this session's process group: the critical bit.
|
||||
setsid nohup bash -c '
|
||||
NAME="$1"; shift
|
||||
"$@" > "'"$JOBS"'/$NAME.out" 2>&1
|
||||
echo $? > "'"$JOBS"'/$NAME.rc"
|
||||
touch "'"$JOBS"'/$NAME.done"
|
||||
' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
|
||||
echo "started $NAME"
|
||||
;;
|
||||
status)
|
||||
# Glob *.out (not *.done) or running jobs are invisible.
|
||||
names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
|
||||
for n in $names; do
|
||||
if [ -f "$JOBS/$n.done" ]; then
|
||||
printf ' %-18s DONE rc=%s %s lines\n' "$n" \
|
||||
"$(cat "$JOBS/$n.rc" 2>/dev/null)" \
|
||||
"$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
elif [ -f "$JOBS/$n.out" ]; then
|
||||
printf ' %-18s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
|
||||
else
|
||||
printf ' %-18s PENDING\n' "$n"
|
||||
fi
|
||||
done
|
||||
;;
|
||||
wait)
|
||||
NAME="$2"; LIMIT="${3:-3600}"; waited=0
|
||||
while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
|
||||
sleep 5; waited=$((waited+5))
|
||||
done
|
||||
[ -f "$JOBS/$NAME.done" ] \
|
||||
&& echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
|
||||
|| echo "TIMEOUT $NAME after ${waited}s"
|
||||
tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
|
||||
;;
|
||||
out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;;
|
||||
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;;
|
||||
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;;
|
||||
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;;
|
||||
esac
|
||||
```
|
||||
|
||||
```bash
|
||||
job.sh start NAME -- <command> # launch detached
|
||||
job.sh status [NAME] # one-shot state
|
||||
job.sh wait NAME 900 25 # block, then print the tail
|
||||
job.sh out NAME 60 # retrieve any past result
|
||||
job.sh tail [NAME] # live follow
|
||||
job.sh watch # refreshing status table
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
- **`.done` is touched LAST**, after the exit code is written — its presence
|
||||
alone means the output file is final.
|
||||
- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running
|
||||
jobs entirely, the exact blindness this fixes.
|
||||
- **Capture the exit code** (`rc=7` proved a self-test worked; without it a
|
||||
job can "complete" having failed).
|
||||
- **Never put the payload in a line you grep for** — the sentinel is the only
|
||||
completion signal.
|
||||
- **Never `pkill -f <pattern>`** where the pattern also matches your own ssh
|
||||
command line — it kills the session. Distinctive names or `pkill -x`.
|
||||
- **Write helper scripts to a file and `scp` them.** Triple-nested
|
||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line);
|
||||
every failure disappeared once the script was a file.
|
||||
|
||||
## Verify before trusting it (3am law)
|
||||
|
||||
```bash
|
||||
job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
|
||||
job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
|
||||
sleep 4; job.sh status # selftest DONE rc=7; slowtest RUN
|
||||
```
|
||||
|
||||
Both paths, including the failure case, and the rc round-trip — a runner that
|
||||
only proves the happy path is not trustworthy.
|
||||
|
||||
## Reporting discipline
|
||||
|
||||
- Give the user a real terminal when the run is long:
|
||||
`alacritty --title "bench - dankserver" -e bash -c 'ssh -o
|
||||
ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &`
|
||||
- State the MECHANISM, not vibes: the job is durable on the host;
|
||||
`job.sh out NAME` retrieves it later. Do not narrate "still loading"
|
||||
repeatedly — check once, act, report the resolved result.
|
||||
- When the user asks for status, one fast command, not a narration.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes
|
||||
- [[Deploy State Triage]] — the two-process deploy this lesson generalizes
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
type: reference
|
||||
title: SSH and Remote Access
|
||||
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
|
||||
tags: [ssh, remote, fish, gui, diagnosis]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# SSH and Remote Access
|
||||
|
||||
## Passwordless SSH bootstrap without sshpass/expect
|
||||
|
||||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||||
flows through the **process env only** — never a file, never argv.
|
||||
|
||||
1. **Generate a key** (if none):
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||||
```
|
||||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||||
helper itself stores NO secret):
|
||||
```bash
|
||||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||||
chmod 700 /tmp/askpass.sh
|
||||
```
|
||||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||||
```bash
|
||||
env DEPLOY_PW='<password>' \
|
||||
SSH_ASKPASS=/tmp/askpass.sh \
|
||||
SSH_ASKPASS_REQUIRE=force \
|
||||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||||
```
|
||||
Password via the transient `env` argument only.
|
||||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||||
BatchMode fails fast if a password would be required; a clean success
|
||||
proves the key works.
|
||||
|
||||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||||
|
||||
## Remote execution on a fish login shell
|
||||
|
||||
Running remote commands or shipping files over SSH to a host whose login
|
||||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||||
workstation).
|
||||
|
||||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||||
— password via env, never argv (ps-visible).
|
||||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||||
echo MISSING; done"'` (escape inner `$`).
|
||||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||||
`md5sum` on both ends.
|
||||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||||
code propagates to the caller anyway.
|
||||
|
||||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||||
|
||||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||||
|
||||
## GUI hang — remote diagnosis over SSH
|
||||
|
||||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||||
|
||||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||||
it:
|
||||
```bash
|
||||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||||
```
|
||||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||||
holder outright).
|
||||
2. **Full stacks** for victim and holder:
|
||||
```bash
|
||||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||||
```
|
||||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||||
not progressing.
|
||||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||||
```bash
|
||||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||||
journalctl -b -k --no-pager | grep -i xid
|
||||
```
|
||||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||||
seconds; then inspect user-session events around it
|
||||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||||
USB workers, etc.
|
||||
|
||||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||||
something that can complete, e.g. storage), recovery may be possible by
|
||||
clearing that dependency first.
|
||||
|
||||
After diagnosing: map stack symbols to driver internals to name the exact
|
||||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: index
|
||||
description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks.
|
||||
tags: [operations, index]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Operations
|
||||
|
||||
Procedures for operating the fleet. The LLM-specific pages live under
|
||||
[LLM](../LLM/index.md); infrastructure and architecture under the
|
||||
`../Infrastructure/` and `../Architecture/` folders.
|
||||
|
||||
- [[Remote Job Runner]] — detached long-job execution with sentinel files;
|
||||
why ssh tail-watchers fail
|
||||
- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk
|
||||
procedure, plus the purge-and-refresh protocol
|
||||
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
|
||||
the pasta port gate
|
||||
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
|
||||
stop-first, pre-commit iteration, incident-window reconstruction, WoL
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
— monitoring config as code
|
||||
- [Deploy State Triage](../Architecture/Deploy%20State%20Triage.md) — the
|
||||
deploy plumbing
|
||||
- [Quadlet Lifecycle](../Architecture/Quadlet%20Lifecycle.md) — the unit layer
|
||||
Reference in new issue
Block a user