Three things the live observability census surfaced, plus the two docs that were lying about them.
frank-archon was un-alertable. radiance (the frank-archon runtime) exports vllm:* metrics, not llamacpp:* — verified on the live endpoint. Every rule in llm-fleet.yml was keyed to llamacpp:*, so archon's queueing / prefix-cache / preemption / time-to-first-token could never fire. Adds a vllm-fleet group: 1 recording (vllm:num_requests_waiting) + 4 alerts — VllmRequestsQueueing, VllmPrefixCacheCollapse, VllmPreempting, VllmTTFTp95.
The observability seam asserted a retired doctrine. It still grepped for a "no metrics for archon" claim with a namespace check that passed vacuously. Rewritten so llm-fleet rules bind llamacpp:*, vllm-fleet rules bind vllm:*, and neither may evaluate the other's namespace.
Loki had no way to be read. Provisioned a Loki datasource beside Prometheus. Loki ships no web UI and the guarded vhost serves only the API — Grafana Explore is the only human surface.
Log pipeline — diagnosed, not fixed (deliberately)
Loki holds zero streams because the shipper ships nothing. The old theory ("wrong log-path glob") is wrong. Proven cause: a rootless container cannot read the host journal — system.journal is root:systemd-journal 0640/adm, other::---, and a rootless busybox cat of it returns DENIED. A second, independent gap: rootless quadlet logs land in the user journal, which the system-only source never reads.
Closing it is a permission/architecture decision (run Alloy as a system service, grant the mapped uid journal read, or journald-forward to a readable path) — not a config edit. Recorded in the Alloy page; left for a deliberate call, not guessed onto a live shipper.
Verification
./tests/run.sh — 471 passed, including the rewritten observability seam.
Live: frank-archon /metrics serves vllm:* (the new rules bind it); Loki reachable and queryable through the new Grafana datasource + Explore.
Out of scope / follow-up
Alloy journal read (above) — needs a deliberate architecture decision.
Loki bind is still LAN 0.0.0.0:3100; tightening to loopback would break the quadlet's loki:3100 push path — separate call.
## What
Three things the live observability census surfaced, plus the two docs that were lying about them.
1. **frank-archon was un-alertable.** radiance (the frank-archon runtime) exports `vllm:*` metrics, not `llamacpp:*` — verified on the live endpoint. Every rule in `llm-fleet.yml` was keyed to `llamacpp:*`, so archon's queueing / prefix-cache / preemption / time-to-first-token could never fire. Adds a `vllm-fleet` group: 1 recording (`vllm:num_requests_waiting`) + 4 alerts — `VllmRequestsQueueing`, `VllmPrefixCacheCollapse`, `VllmPreempting`, `VllmTTFTp95`.
2. **The observability seam asserted a retired doctrine.** It still grepped for a "no metrics for archon" claim with a namespace check that passed vacuously. Rewritten so `llm-fleet` rules bind `llamacpp:*`, `vllm-fleet` rules bind `vllm:*`, and neither may evaluate the other's namespace.
3. **Loki had no way to be read.** Provisioned a Loki datasource beside Prometheus. Loki ships no web UI and the guarded vhost serves only the API — Grafana Explore is the only human surface.
## Log pipeline — diagnosed, not fixed (deliberately)
Loki holds zero streams because the shipper ships nothing. The old theory ("wrong log-path glob") is **wrong**. Proven cause: a rootless container cannot read the host journal — `system.journal` is `root:systemd-journal 0640`/`adm`, `other::---`, and a rootless `busybox cat` of it returns **DENIED**. A second, independent gap: rootless quadlet logs land in the *user* journal, which the system-only source never reads.
Closing it is a permission/architecture decision (run Alloy as a system service, grant the mapped uid journal read, or journald-forward to a readable path) — not a config edit. Recorded in the Alloy page; **left for a deliberate call**, not guessed onto a live shipper.
## Verification
- `./tests/run.sh` — 471 passed, including the rewritten observability seam.
- Live: frank-archon `/metrics` serves `vllm:*` (the new rules bind it); Loki reachable and queryable through the new Grafana datasource + Explore.
## Out of scope / follow-up
- Alloy journal read (above) — needs a deliberate architecture decision.
- Loki bind is still LAN `0.0.0.0:3100`; tightening to loopback would break the quadlet's `loki:3100` push path — separate call.
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.
Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
dank
merged commit 2fe68e69af into main2026-10-20 19:39:38 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What
Three things the live observability census surfaced, plus the two docs that were lying about them.
vllm:*metrics, notllamacpp:*— verified on the live endpoint. Every rule inllm-fleet.ymlwas keyed tollamacpp:*, so archon's queueing / prefix-cache / preemption / time-to-first-token could never fire. Adds avllm-fleetgroup: 1 recording (vllm:num_requests_waiting) + 4 alerts —VllmRequestsQueueing,VllmPrefixCacheCollapse,VllmPreempting,VllmTTFTp95.llm-fleetrules bindllamacpp:*,vllm-fleetrules bindvllm:*, and neither may evaluate the other's namespace.Log pipeline — diagnosed, not fixed (deliberately)
Loki holds zero streams because the shipper ships nothing. The old theory ("wrong log-path glob") is wrong. Proven cause: a rootless container cannot read the host journal —
system.journalisroot:systemd-journal 0640/adm,other::---, and a rootlessbusybox catof it returns DENIED. A second, independent gap: rootless quadlet logs land in the user journal, which the system-only source never reads.Closing it is a permission/architecture decision (run Alloy as a system service, grant the mapped uid journal read, or journald-forward to a readable path) — not a config edit. Recorded in the Alloy page; left for a deliberate call, not guessed onto a live shipper.
Verification
./tests/run.sh— 471 passed, including the rewritten observability seam./metricsservesvllm:*(the new rules bind it); Loki reachable and queryable through the new Grafana datasource + Explore.Out of scope / follow-up
0.0.0.0:3100; tightening to loopback would break the quadlet'sloki:3100push path — separate call.cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15b5c-ede9-7063-b05b-1ef94188b7c3