---
name: rootless-to-root-quadlet-journal-read
description: "Convert a rootless Podman Quadlet (e.g. Alloy) to a ROOT system service so it can read the host journal — the machine-id mount, Network=host for loopback-published Loki, root image mirror, and the user-unit cutover. Use when an sd_journal/loki.source.journal reader ships zero lines with a green health check."
---

# Rootless→root Quadlet so a container can read the host journal

Symptom: a rootless Quadlet that scrapes the systemd journal (`loki.source.journal`,
any `sd_journal` reader) ships **zero** lines — silently, `health` green,
`loki_source_journal_*_lines_total 0`, no error in the log.

## Why (proven)
- Journal files are `root:systemd-journal 0640` (`group:adm`, `other::---`). A rootless
  container spawns with no supplementary groups → `sd_journal` opens an empty view.
  Proof a rootless read is denied: `podman run --rm -v /var/log/journal:...:ro busybox
  cat /var/log/journal/*/*.journal` → `DENIED`.
- Rootless Quadlet **container** logs also live in `user-<uid>.journal` (journald log
  driver), not `ctr.log`. A system-journal-only read misses them.

## The shape that works (root, not rootless)
Run it as a **root system Quadlet** (`/etc/containers/systemd/<svc>.container`,
`WantedBy=multi-user.target`). Root's DAC override opens every journal file, so one
`loki.source.journal` reads `system.journal` **and** `user-*.journal` together.

Four load-bearing bits (all validated on Dankclaw Alloy 2026-10-20):
1. **`Volume=/etc/machine-id:/etc/machine-id:ro`** — the single biggest gotcha.
   `sd_journal` locates the journal dir by the *local* machine id; without it the
   container opens **nothing** and reads exactly 0 lines, health green. A root probe
   without it = 0; with it = 31,825 lines in 20s.
2. **`Network=host`** — if the sink (Loki) publishes `127.0.0.1:3100` (loopback), no
   bridge container (root or rootless) can reach the host loopback; `host.containers.internal`
   and the LAN IP are both refused. Host network + push to `http://127.0.0.1:3100`.
   Side effect: bind the app UI to `127.0.0.1` (no `PublishPort`) so it stays loopback-only.
3. **`/var/log/journal` + `/run/log/journal`** mounted `:ro`.
4. **Image must be in the ROOT store.** A root `podman build` runs in the build
   container's own resolver and fails DNS here. Build rootless, then mirror:
   `podman save -o t.tar <ref> && sudo podman load -i t.tar`. Compare image ids so a
   rebuild actually re-lands (stale root tag is a silent footgun).

## Deployer cutover (retire the rootless unit FIRST)
Both the user and system managers generate `<svc>.service`; two containers named
`<svc>` both binding the same loopback port cannot coexist. Before starting the system
unit: `systemctl --user stop <svc>.service`; `mv` the user quadlet out of
`~/.config/containers/systemd/` (it regenerates from there) to `~/.AGENTTRASH/<date>-<svc>-root-cutover/`;
`systemctl --user daemon-reload`. Then install to `/etc/containers/systemd/`,
`sudo systemctl daemon-reload`, `sudo systemctl restart <svc>.service`. Generated
(system) units cannot be `enable`d here — boot persistence is the `[Install]` section.

## Probe the exact command BEFORE writing the unit
`sudo podman run --rm --net=host -v /etc/machine-id:...:ro -v journal mounts
  <image> run --server.http.listen-addr=127.0.0.1:<free-port> <config>` — pick a FREE UI
port (the deployed instance already owns the real one; a bind conflict reads as
"reads 0 lines", not "port in use"). A squashed one-line River/HCL block is invalid —
attributes need newline terminators, or the probe dies before reading anything.

## Verify (query gotchas that caused false "empty" reads)
- `loki_source_journal_target_lines_total` must climb.
- Loki `/loki/api/v1/query_range` returns `values` as `["<ts_ns>","<line>"]` **arrays** —
  grep `"ts"` finds nothing; the data is there. `/series` defaults to a short index window
  and can read empty while recent data is queryable. Use a wide `start`/`end`, no `direction`.
- A fresh journal reader replays the whole persistent journal from the oldest; Loki
  `reject_old_samples` + the per-tenant rate limit discard old/throttled entries
  (`loki_discarded_samples_total{reason="rate_limited"}`) without counting as writer
  drops. It self-clears once it reaches the tail; raise the ingestion rate to speed it or
  start at the head to skip.