0 Fork 0

Files

T

Frank 8af40c402b chore(agent): commit the machine tier and the doctrine text the checkout was already running

Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and
`merge --ff-only` refuses while tracked paths are dirty. Discard is not available here
(workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed.

- `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already
  byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new
  prose — the commit exists so the trunk merge can proceed.
- `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat
  owns (AGENTS.md § Live-sync tiers).

`ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the
only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the
inert all-zero AWS example id — and they appear inside the two skill pages that teach
that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged
gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal
dies with this landing. No `--no-verify` was used: the gate ran and its verdict is
recorded here.

Refs #61

2026-10-21 01:44:51 -07:00

4.3 KiB

Raw Blame History

name, description

name	description
rootless-to-root-quadlet-journal-read	Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check.

Rootless→root Quadlet so a container can read the host journal

Symptom: a rootless Quadlet that scrapes the systemd journal (loki.source.journal, any sd_journal reader) ships zero lines — silently, health green, loki_source_journal_*_lines_total 0, no error in the log.

Why (proven)

Journal files are root:systemd-journal 0640 (group:adm, other::---). A rootless container spawns with no supplementary groups → sd_journal opens an empty view. Proof a rootless read is denied: podman run --rm -v /var/log/journal:...:ro busybox cat /var/log/journal/*/*.journal → DENIED.
Rootless Quadlet container logs also live in user-<uid>.journal (journald log driver), not ctr.log. A system-journal-only read misses them.

The shape that works (root, not rootless)

Run it as a root system Quadlet (/etc/containers/systemd/<svc>.container, WantedBy=multi-user.target). Root's DAC override opens every journal file, so one loki.source.journal reads system.journal and user-*.journal together.

Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):

Volume=/etc/machine-id:/etc/machine-id:ro — the single biggest gotcha. sd_journal locates the journal dir by the local machine id; without it the container opens nothing and reads exactly 0 lines, health green. A root probe without it = 0; with it = 31,825 lines in 20s.
Network=host — if the sink (Loki) publishes 127.0.0.1:3100 (loopback), no bridge container (root or rootless) can reach the host loopback; host.containers.internal and the LAN IP are both refused. Host network + push to http://127.0.0.1:3100. Side effect: bind the app UI to 127.0.0.1 (no PublishPort) so it stays loopback-only.
/var/log/journal + /run/log/journal mounted :ro.
Image must be in the ROOT store. A root podman build runs in the build container's own resolver and fails DNS here. Build rootless, then mirror: podman save -o t.tar <ref> && sudo podman load -i t.tar. Compare image ids so a rebuild actually re-lands (stale root tag is a silent footgun).

Deployer cutover (retire the rootless unit FIRST)

Both the user and system managers generate <svc>.service; two containers named <svc> both binding the same loopback port cannot coexist. Before starting the system unit: systemctl --user stop <svc>.service; mv the user quadlet out of ~/.config/containers/systemd/ (it regenerates from there) to ~/.AGENTTRASH/<date>-<svc>-root-cutover/; systemctl --user daemon-reload. Then install to /etc/containers/systemd/, sudo systemctl daemon-reload, sudo systemctl restart <svc>.service. Generated (system) units cannot be enabled here — boot persistence is the [Install] section.

Probe the exact command BEFORE writing the unit

sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts <image> run --server.http.listen-addr=127.0.0.1:<free-port> <config> — pick a FREE UI port (the deployed instance already owns the real one; a bind conflict reads as "reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid — attributes need newline terminators, or the probe dies before reading anything.

Verify (query gotchas that caused false "empty" reads)

loki_source_journal_target_lines_total must climb.
Loki /loki/api/v1/query_range returns values as ["<ts_ns>","<line>"] arrays — grep "ts" finds nothing; the data is there. /series defaults to a short index window and can read empty while recent data is queryable. Use a wide start/end, no direction.
A fresh journal reader replays the whole persistent journal from the oldest; Loki reject_old_samples + the per-tenant rate limit discard old/throttled entries (loki_discarded_samples_total{reason="rate_limited"}) without counting as writer drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or start at the head to skip.

4.3 KiB Raw Blame History