The log pipeline was provisioned but empty: Alloy shipped zero lines and Loki held zero streams, with a green health check hiding it. This makes Alloy actually deliver.
Root cause (proven on the host, not guessed)
Two independent reasons a rootless Alloy reads nothing:
The journal files are root:systemd-journal 0640 (other::---); a rootless container has no supplementary groups, so opening them returns DENIED. loki.source.journal then sees an empty view silently.
Rootless Quadlet container logs land in the user journal (user-1000.journal), and the old container-file globs point at a root-storage ctr.log layout that does not exist here.
The fix — Alloy becomes a root system service on Network=host
Root reads every journal file under the machine-id dir — system.journalanduser-*.journal — so one source carries host services and the rootless containers' logs.
/etc/machine-id mounted ro is load-bearing: sd_journal locates the journal by the local machine id; without it the source reads exactly 0 lines, silently, health green. (This is what hid the bug.)
Network=host: Loki publishes 127.0.0.1:3100 (loopback only, unchanged), and no bridge container can reach host loopback — so Alloy shares the host network and pushes to 127.0.0.1:3100. The UI binds 127.0.0.1:12345 directly (no PublishPort), so it stays loopback-only.
Deployer: new deploy_alloy routes around the generic rootless loop — builds rootless, mirrors the image into the root store (a root build fails on DNS in this host's build container), retires the old rootless alloy unit (both managers generate alloy.service; two containers named alloy can't coexist), installs to /etc/containers/systemd/.
Validation (throwaway root probe, before writing the unit)
A --net=host root container with the mounts + loopback-push config: 31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero drops, Loki's label set went empty → populated. The same probe without/etc/machine-id read 0 — isolating the load-bearing mount.
Test seam
Rewrote the assertions that encoded alloy-as-rootless (boot target multi-user.target, Network=host, loopback push URL, no cross-manager loki.service dep, loopback UI) and added a test pinning the load-bearing properties: Network=host, the /etc/machine-id mount, the journal mounts, and the system-install + root-mirror deployer path. ./tests/run.sh green.
Deploy-pending
Built and tested in-repo; the live instance is still the rootless unit until ./scripts/deploy.sh dankclaw --service alloy runs (Frank's window). The cutover retires the rootless unit and its quadlet file to ~/.AGENTTRASH/<date>-alloy-root-cutover/.
## What
The log pipeline was **provisioned but empty**: Alloy shipped zero lines and Loki held zero streams, with a green health check hiding it. This makes Alloy actually deliver.
## Root cause (proven on the host, not guessed)
Two independent reasons a rootless Alloy reads nothing:
1. The journal files are `root:systemd-journal 0640` (`other::---`); a rootless container has no supplementary groups, so opening them returns **DENIED**. `loki.source.journal` then sees an empty view silently.
2. Rootless Quadlet container logs land in the **user** journal (`user-1000.journal`), and the old container-file globs point at a root-storage `ctr.log` layout that does not exist here.
## The fix — Alloy becomes a root system service on `Network=host`
- **Root** reads every journal file under the machine-id dir — `system.journal` *and* `user-*.journal` — so one source carries host services *and* the rootless containers' logs.
- **`/etc/machine-id` mounted ro is load-bearing**: `sd_journal` locates the journal by the local machine id; without it the source reads exactly 0 lines, silently, health green. (This is what hid the bug.)
- **`Network=host`**: Loki publishes `127.0.0.1:3100` (loopback only, unchanged), and no bridge container can reach host loopback — so Alloy shares the host network and pushes to `127.0.0.1:3100`. The UI binds `127.0.0.1:12345` directly (no `PublishPort`), so it stays loopback-only.
- Deployer: new `deploy_alloy` routes around the generic rootless loop — builds rootless, **mirrors the image into the root store** (a root build fails on DNS in this host's build container), retires the old rootless `alloy` unit (both managers generate `alloy.service`; two containers named `alloy` can't coexist), installs to `/etc/containers/systemd/`.
## Validation (throwaway root probe, before writing the unit)
A `--net=host` root container with the mounts + loopback-push config: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero drops**, Loki's label set went empty → populated. The same probe *without* `/etc/machine-id` read 0 — isolating the load-bearing mount.
## Test seam
Rewrote the assertions that encoded alloy-as-rootless (boot target `multi-user.target`, `Network=host`, loopback push URL, no cross-manager `loki.service` dep, loopback UI) and added a test pinning the load-bearing properties: `Network=host`, the `/etc/machine-id` mount, the journal mounts, and the system-install + root-mirror deployer path. `./tests/run.sh` green.
## Deploy-pending
Built and tested in-repo; the **live** instance is still the rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs (Frank's window). The cutover retires the rootless unit and its quadlet file to `~/.AGENTTRASH/<date>-alloy-root-cutover/`.
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.
As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.
deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
dank
merged commit 980491668c into main2026-10-20 20:05:00 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
The log pipeline was provisioned but empty: Alloy shipped zero lines and Loki held zero streams, with a green health check hiding it. This makes Alloy actually deliver.
Root cause (proven on the host, not guessed)
Two independent reasons a rootless Alloy reads nothing:
root:systemd-journal 0640(other::---); a rootless container has no supplementary groups, so opening them returns DENIED.loki.source.journalthen sees an empty view silently.user-1000.journal), and the old container-file globs point at a root-storagectr.loglayout that does not exist here.The fix — Alloy becomes a root system service on
Network=hostsystem.journalanduser-*.journal— so one source carries host services and the rootless containers' logs./etc/machine-idmounted ro is load-bearing:sd_journallocates the journal by the local machine id; without it the source reads exactly 0 lines, silently, health green. (This is what hid the bug.)Network=host: Loki publishes127.0.0.1:3100(loopback only, unchanged), and no bridge container can reach host loopback — so Alloy shares the host network and pushes to127.0.0.1:3100. The UI binds127.0.0.1:12345directly (noPublishPort), so it stays loopback-only.deploy_alloyroutes around the generic rootless loop — builds rootless, mirrors the image into the root store (a root build fails on DNS in this host's build container), retires the old rootlessalloyunit (both managers generatealloy.service; two containers namedalloycan't coexist), installs to/etc/containers/systemd/.Validation (throwaway root probe, before writing the unit)
A
--net=hostroot container with the mounts + loopback-push config: 31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero drops, Loki's label set went empty → populated. The same probe without/etc/machine-idread 0 — isolating the load-bearing mount.Test seam
Rewrote the assertions that encoded alloy-as-rootless (boot target
multi-user.target,Network=host, loopback push URL, no cross-managerloki.servicedep, loopback UI) and added a test pinning the load-bearing properties:Network=host, the/etc/machine-idmount, the journal mounts, and the system-install + root-mirror deployer path../tests/run.shgreen.Deploy-pending
Built and tested in-repo; the live instance is still the rootless unit until
./scripts/deploy.sh dankclaw --service alloyruns (Frank's window). The cutover retires the rootless unit and its quadlet file to~/.AGENTTRASH/<date>-alloy-root-cutover/.cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3