chore(agent): commit the machine tier and the doctrine text the checkout was already running
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
This commit is contained in:
1 parent
004962ffce
commit
8af40c402b
79 files changed
+2771
-332
No files matched your search
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: rootless-to-root-quadlet-journal-read
|
||||
description: "Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check."
|
||||
---
|
||||
|
||||
# Rootless→root Quadlet so a container can read the host journal
|
||||
|
||||
Symptom: a rootless Quadlet that scrapes the systemd journal (`loki.source.journal`,
|
||||
any `sd_journal` reader) ships **zero** lines — silently, `health` green,
|
||||
`loki_source_journal_*_lines_total 0`, no error in the log.
|
||||
|
||||
## Why (proven)
|
||||
- Journal files are `root:systemd-journal 0640` (`group:adm`, `other::---`). A rootless
|
||||
container spawns with no supplementary groups → `sd_journal` opens an empty view.
|
||||
Proof a rootless read is denied: `podman run --rm -v /var/log/journal:...:ro busybox
|
||||
cat /var/log/journal/*/*.journal` → `DENIED`.
|
||||
- Rootless Quadlet **container** logs also live in `user-<uid>.journal` (journald log
|
||||
driver), not `ctr.log`. A system-journal-only read misses them.
|
||||
|
||||
## The shape that works (root, not rootless)
|
||||
Run it as a **root system Quadlet** (`/etc/containers/systemd/<svc>.container`,
|
||||
`WantedBy=multi-user.target`). Root's DAC override opens every journal file, so one
|
||||
`loki.source.journal` reads `system.journal` **and** `user-*.journal` together.
|
||||
|
||||
Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):
|
||||
1. **`Volume=/etc/machine-id:/etc/machine-id:ro`** — the single biggest gotcha.
|
||||
`sd_journal` locates the journal dir by the *local* machine id; without it the
|
||||
container opens **nothing** and reads exactly 0 lines, health green. A root probe
|
||||
without it = 0; with it = 31,825 lines in 20s.
|
||||
2. **`Network=host`** — if the sink (Loki) publishes `127.0.0.1:3100` (loopback), no
|
||||
bridge container (root or rootless) can reach the host loopback; `host.containers.internal`
|
||||
and the LAN IP are both refused. Host network + push to `http://127.0.0.1:3100`.
|
||||
Side effect: bind the app UI to `127.0.0.1` (no `PublishPort`) so it stays loopback-only.
|
||||
3. **`/var/log/journal` + `/run/log/journal`** mounted `:ro`.
|
||||
4. **Image must be in the ROOT store.** A root `podman build` runs in the build
|
||||
container's own resolver and fails DNS here. Build rootless, then mirror:
|
||||
`podman save -o t.tar <ref> && sudo podman load -i t.tar`. Compare image ids so a
|
||||
rebuild actually re-lands (stale root tag is a silent footgun).
|
||||
|
||||
## Deployer cutover (retire the rootless unit FIRST)
|
||||
Both the user and system managers generate `<svc>.service`; two containers named
|
||||
`<svc>` both binding the same loopback port cannot coexist. Before starting the system
|
||||
unit: `systemctl --user stop <svc>.service`; `mv` the user quadlet out of
|
||||
`~/.config/containers/systemd/` (it regenerates from there) to `~/.AGENTTRASH/<date>-<svc>-root-cutover/`;
|
||||
`systemctl --user daemon-reload`. Then install to `/etc/containers/systemd/`,
|
||||
`sudo systemctl daemon-reload`, `sudo systemctl restart <svc>.service`. Generated
|
||||
(system) units cannot be `enable`d here — boot persistence is the `[Install]` section.
|
||||
|
||||
## Probe the exact command BEFORE writing the unit
|
||||
`sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts
|
||||
<image> run --server.http.listen-addr=127.0.0.1:<free-port> <config>` — pick a FREE UI
|
||||
port (the deployed instance already owns the real one; a bind conflict reads as
|
||||
"reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid —
|
||||
attributes need newline terminators, or the probe dies before reading anything.
|
||||
|
||||
## Verify (query gotchas that caused false "empty" reads)
|
||||
- `loki_source_journal_target_lines_total` must climb.
|
||||
- Loki `/loki/api/v1/query_range` returns `values` as `["<ts_ns>","<line>"]` **arrays** —
|
||||
grep `"ts"` finds nothing; the data is there. `/series` defaults to a short index window
|
||||
and can read empty while recent data is queryable. Use a wide `start`/`end`, no `direction`.
|
||||
- A fresh journal reader replays the whole persistent journal from the oldest; Loki
|
||||
`reject_old_samples` + the per-tenant rate limit discard old/throttled entries
|
||||
(`loki_discarded_samples_total{reason="rate_limited"}`) without counting as writer
|
||||
drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or
|
||||
start at the head to skip.
|
||||
Reference in new issue
Block a user