Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
66 lines
4.3 KiB
Markdown
66 lines
4.3 KiB
Markdown
---
|
|
name: rootless-to-root-quadlet-journal-read
|
|
description: "Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check."
|
|
---
|
|
|
|
# Rootless→root Quadlet so a container can read the host journal
|
|
|
|
Symptom: a rootless Quadlet that scrapes the systemd journal (`loki.source.journal`,
|
|
any `sd_journal` reader) ships **zero** lines — silently, `health` green,
|
|
`loki_source_journal_*_lines_total 0`, no error in the log.
|
|
|
|
## Why (proven)
|
|
- Journal files are `root:systemd-journal 0640` (`group:adm`, `other::---`). A rootless
|
|
container spawns with no supplementary groups → `sd_journal` opens an empty view.
|
|
Proof a rootless read is denied: `podman run --rm -v /var/log/journal:...:ro busybox
|
|
cat /var/log/journal/*/*.journal` → `DENIED`.
|
|
- Rootless Quadlet **container** logs also live in `user-<uid>.journal` (journald log
|
|
driver), not `ctr.log`. A system-journal-only read misses them.
|
|
|
|
## The shape that works (root, not rootless)
|
|
Run it as a **root system Quadlet** (`/etc/containers/systemd/<svc>.container`,
|
|
`WantedBy=multi-user.target`). Root's DAC override opens every journal file, so one
|
|
`loki.source.journal` reads `system.journal` **and** `user-*.journal` together.
|
|
|
|
Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):
|
|
1. **`Volume=/etc/machine-id:/etc/machine-id:ro`** — the single biggest gotcha.
|
|
`sd_journal` locates the journal dir by the *local* machine id; without it the
|
|
container opens **nothing** and reads exactly 0 lines, health green. A root probe
|
|
without it = 0; with it = 31,825 lines in 20s.
|
|
2. **`Network=host`** — if the sink (Loki) publishes `127.0.0.1:3100` (loopback), no
|
|
bridge container (root or rootless) can reach the host loopback; `host.containers.internal`
|
|
and the LAN IP are both refused. Host network + push to `http://127.0.0.1:3100`.
|
|
Side effect: bind the app UI to `127.0.0.1` (no `PublishPort`) so it stays loopback-only.
|
|
3. **`/var/log/journal` + `/run/log/journal`** mounted `:ro`.
|
|
4. **Image must be in the ROOT store.** A root `podman build` runs in the build
|
|
container's own resolver and fails DNS here. Build rootless, then mirror:
|
|
`podman save -o t.tar <ref> && sudo podman load -i t.tar`. Compare image ids so a
|
|
rebuild actually re-lands (stale root tag is a silent footgun).
|
|
|
|
## Deployer cutover (retire the rootless unit FIRST)
|
|
Both the user and system managers generate `<svc>.service`; two containers named
|
|
`<svc>` both binding the same loopback port cannot coexist. Before starting the system
|
|
unit: `systemctl --user stop <svc>.service`; `mv` the user quadlet out of
|
|
`~/.config/containers/systemd/` (it regenerates from there) to `~/.AGENTTRASH/<date>-<svc>-root-cutover/`;
|
|
`systemctl --user daemon-reload`. Then install to `/etc/containers/systemd/`,
|
|
`sudo systemctl daemon-reload`, `sudo systemctl restart <svc>.service`. Generated
|
|
(system) units cannot be `enable`d here — boot persistence is the `[Install]` section.
|
|
|
|
## Probe the exact command BEFORE writing the unit
|
|
`sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts
|
|
<image> run --server.http.listen-addr=127.0.0.1:<free-port> <config>` — pick a FREE UI
|
|
port (the deployed instance already owns the real one; a bind conflict reads as
|
|
"reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid —
|
|
attributes need newline terminators, or the probe dies before reading anything.
|
|
|
|
## Verify (query gotchas that caused false "empty" reads)
|
|
- `loki_source_journal_target_lines_total` must climb.
|
|
- Loki `/loki/api/v1/query_range` returns `values` as `["<ts_ns>","<line>"]` **arrays** —
|
|
grep `"ts"` finds nothing; the data is there. `/series` defaults to a short index window
|
|
and can read empty while recent data is queryable. Use a wide `start`/`end`, no `direction`.
|
|
- A fresh journal reader replays the whole persistent journal from the oldest; Loki
|
|
`reject_old_samples` + the per-tenant rate limit discard old/throttled entries
|
|
(`loki_discarded_samples_total{reason="rate_limited"}`) without counting as writer
|
|
drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or
|
|
start at the head to skip.
|