chore(agent): commit the machine tier and the doctrine text the checkout was already running

Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and
`merge --ff-only` refuses while tracked paths are dirty. Discard is not available here
(workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed.

- `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already
  byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new
  prose — the commit exists so the trunk merge can proceed.
- `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat
  owns (AGENTS.md § Live-sync tiers).

`ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the
only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the
inert all-zero AWS example id — and they appear inside the two skill pages that teach
that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged
gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal
dies with this landing. No `--no-verify` was used: the gate ran and its verdict is
recorded here.

Refs #61
This commit is contained in:
Frank committed 2026-10-21 01:44:51 -07:00
1 parent 004962ffce
commit 8af40c402b
79 files changed
+2771 -332

No files matched your search

@@ -0,0 +1,65 @@
---
name: rootless-to-root-quadlet-journal-read
description: "Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check."
---
# Rootless→root Quadlet so a container can read the host journal
Symptom: a rootless Quadlet that scrapes the systemd journal (`loki.source.journal`,
any `sd_journal` reader) ships **zero** lines — silently, `health` green,
`loki_source_journal_*_lines_total 0`, no error in the log.
## Why (proven)
- Journal files are `root:systemd-journal 0640` (`group:adm`, `other::---`). A rootless
container spawns with no supplementary groups → `sd_journal` opens an empty view.
Proof a rootless read is denied: `podman run --rm -v /var/log/journal:...:ro busybox
cat /var/log/journal/*/*.journal` → `DENIED`.
- Rootless Quadlet **container** logs also live in `user-<uid>.journal` (journald log
driver), not `ctr.log`. A system-journal-only read misses them.
## The shape that works (root, not rootless)
Run it as a **root system Quadlet** (`/etc/containers/systemd/<svc>.container`,
`WantedBy=multi-user.target`). Root's DAC override opens every journal file, so one
`loki.source.journal` reads `system.journal` **and** `user-*.journal` together.
Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):
1. **`Volume=/etc/machine-id:/etc/machine-id:ro`** — the single biggest gotcha.
`sd_journal` locates the journal dir by the *local* machine id; without it the
container opens **nothing** and reads exactly 0 lines, health green. A root probe
without it = 0; with it = 31,825 lines in 20s.
2. **`Network=host`** — if the sink (Loki) publishes `127.0.0.1:3100` (loopback), no
bridge container (root or rootless) can reach the host loopback; `host.containers.internal`
and the LAN IP are both refused. Host network + push to `http://127.0.0.1:3100`.
Side effect: bind the app UI to `127.0.0.1` (no `PublishPort`) so it stays loopback-only.
3. **`/var/log/journal` + `/run/log/journal`** mounted `:ro`.
4. **Image must be in the ROOT store.** A root `podman build` runs in the build
container's own resolver and fails DNS here. Build rootless, then mirror:
`podman save -o t.tar <ref> && sudo podman load -i t.tar`. Compare image ids so a
rebuild actually re-lands (stale root tag is a silent footgun).
## Deployer cutover (retire the rootless unit FIRST)
Both the user and system managers generate `<svc>.service`; two containers named
`<svc>` both binding the same loopback port cannot coexist. Before starting the system
unit: `systemctl --user stop <svc>.service`; `mv` the user quadlet out of
`~/.config/containers/systemd/` (it regenerates from there) to `~/.AGENTTRASH/<date>-<svc>-root-cutover/`;
`systemctl --user daemon-reload`. Then install to `/etc/containers/systemd/`,
`sudo systemctl daemon-reload`, `sudo systemctl restart <svc>.service`. Generated
(system) units cannot be `enable`d here — boot persistence is the `[Install]` section.
## Probe the exact command BEFORE writing the unit
`sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts
<image> run --server.http.listen-addr=127.0.0.1:<free-port> <config>` — pick a FREE UI
port (the deployed instance already owns the real one; a bind conflict reads as
"reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid —
attributes need newline terminators, or the probe dies before reading anything.
## Verify (query gotchas that caused false "empty" reads)
- `loki_source_journal_target_lines_total` must climb.
- Loki `/loki/api/v1/query_range` returns `values` as `["<ts_ns>","<line>"]` **arrays** —
grep `"ts"` finds nothing; the data is there. `/series` defaults to a short index window
and can read empty while recent data is queryable. Use a wide `start`/`end`, no `direction`.
- A fresh journal reader replays the whole persistent journal from the oldest; Loki
`reject_old_samples` + the per-tenant rate limit discard old/throttled entries
(`loki_discarded_samples_total{reason="rate_limited"}`) without counting as writer
drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or
start at the head to skip.