From eb4d9910cbbc75aa1167f5283f582e8c6fe7fd8f Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 19:28:20 -0700 Subject: [PATCH 1/3] feat(observability): add vllm-fleet rule group for frank-archon radiance exports vllm:* metrics (verified on the live endpoint), not llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache collapse, preemption, p95 time-to-first-token) keyed to vllm:*. Split the observability seam so llm-fleet rules bind llamacpp:* and vllm-fleet rules bind vllm:*, and neither may evaluate the other's namespace. Drops the retired no-metrics-for-archon doctrine the seam still asserted and the vacuous namespace grep. --- shared/prometheus/rules/llm-fleet.yml | 102 +++++++++++++++++++++++--- tests/observability_quadlet.bats | 90 +++++++++++++---------- 2 files changed, 146 insertions(+), 46 deletions(-) diff --git a/shared/prometheus/rules/llm-fleet.yml b/shared/prometheus/rules/llm-fleet.yml index e843b68..4249ae1 100644 --- a/shared/prometheus/rules/llm-fleet.yml +++ b/shared/prometheus/rules/llm-fleet.yml @@ -1,10 +1,15 @@ # Alerting and recording rules for the LLM fleet. # -# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both -# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16 -# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the -# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was -# deleted rather than left never-green. +# Two metric namespaces, because two runtime families are scraped: +# +# * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which +# re-exports the llama.cpp metric names). Verified from /metrics on +# 2026-09-16 / 2026-09-20. +# * vllm:* — frank-archon, the radiance engine on Dankserver since +# 2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so +# it is a separate scrape job in prometheus.yml and the rules below are a +# separate group. (The old claim that its scrape job was deleted is false: +# it re-joined the day it left the FreeToken engine and is `up` today.) # (The netavark published-port quirk that forces the Dankclaw endpoints # onto the gateway is same-host only.) # @@ -22,7 +27,8 @@ # cached by cached + prompt. # * Speculative decoding: drafts and accepted tokens are both counters, # so LlmSpecDecodeOff watches the pair. -# * No latency histograms exist, so there is no p95 alert. +# * No latency histograms in llama.cpp, so no p95 alert there; radiance's +# vllm:* namespace DOES export one, so the vllm-fleet group alerts on it. # # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/ @@ -112,6 +118,84 @@ groups: description: >- Prometheus cannot reach the LLM endpoint. frank-zealot and frank-dragoon are root units on Dankclaw: `systemctl is-active - frank-zealot` / `frank-dragoon`. Both scraped endpoints export - the llamacpp:* family. (frank-archon left the scrape set - 2026-10-05 — FreeToken exports no /metrics.) + frank-zealot` / `frank-dragoon` (both export llamacpp:* — + dragoon via halogen's llama.cpp-compatible names). frank-archon + is a radiance endpoint on Dankserver (`systemctl show + frank-archon` there) exporting the vllm:* family; the + `frank-.*` selector covers all three regardless of namespace. + + # frank-archon (radiance) exports the vLLM-compatible namespace, NOT + # llamacpp:*. Every rule here is keyed to a series the exporter actually + # emits (verified from /metrics 2026-10-20: num_requests_waiting, + # gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total, + # time_to_first_token_seconds). The llamacpp rules above evaluate nothing for + # this job and these evaluate nothing for llama.cpp/halogen — two clean + # namespaces, each keyed only to what its runtime exports. + - name: vllm-fleet + interval: 30s + rules: + # p95 time-to-first-token, shared with the dashboard. When the endpoint + # is idle every bucket rate is zero, histogram_quantile yields NaN, and + # neither this nor the alert below can fire on an idle engine. + - record: job:vllm_ttft_p95:rate5m + expr: | + histogram_quantile(0.95, sum by (job, le) + (rate(vllm:time_to_first_token_seconds_bucket[5m]))) + + # Radiance exports a real queue-depth gauge (llama.cpp only shows the + # wall as requests_deferred). --max-num-seqs is small, so a sustained + # wait is either real load or this endpoint carrying the fleet alone. + - alert: VllmRequestsQueueing + expr: vllm:num_requests_waiting > 0 + for: 10m + labels: + severity: warning + annotations: + summary: "{{ $labels.job }}: requests waiting for 10 minutes" + description: >- + The radiance scheduler has requests queued. Either the fleet is + genuinely busy or this endpoint is carrying it alone. + + # Radiance reports prefix reuse directly as a hit-rate gauge — no + # counter division needed. The token-rate floor keeps an idle engine + # (hit rate 0 with no traffic) from firing. + - alert: VllmPrefixCacheCollapse + expr: >- + vllm:gpu_prefix_cache_hit_rate < 0.20 + and rate(vllm:prompt_tokens_total[5m]) > 0.05 + for: 15m + labels: + severity: info + annotations: + summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes" + description: >- + Nearly every request prefilling from scratch. With several large + contexts in flight this is the capacity wall, not a routing fault. + + # Preemption is the KV-thrash signature radiance reports that llama.cpp + # cannot: the engine evicts a running request to admit another. Any + # sustained rate means the KV cache cannot hold the concurrency in flight. + - alert: VllmPreempting + expr: rate(vllm:num_preemptions_total[5m]) > 0 + for: 5m + labels: + severity: warning + annotations: + summary: "{{ $labels.job }}: engine preempting requests" + description: >- + The engine is evicting running requests to admit new ones — the KV + cache is undersized for the concurrency in flight. + + # p95 first-token latency, kept HIGH on purpose: a large-prompt prefill + # legitimately spends many seconds before token 1, so this fires only on + # a wedged scheduler or a queue backed up behind one request. + - alert: VllmTtftSlow + expr: job:vllm_ttft_p95:rate5m > 120 + for: 10m + labels: + severity: warning + annotations: + summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s" + description: >- + First-token latency is pathological, not an honest big prefill. + Likely a wedged scheduler or one request stalling the queue. diff --git a/tests/observability_quadlet.bats b/tests/observability_quadlet.bats index ff2d312..12119cd 100755 --- a/tests/observability_quadlet.bats +++ b/tests/observability_quadlet.bats @@ -313,33 +313,33 @@ PAIRS done } -@test "prometheus scrapes every LLM endpoint that exports metrics" { - # The LLM endpoints are the services whose behaviour this fleet argues about - # most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw - # endpoints export the llamacpp:* family. frank-archon LEFT the set - # 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint — - # scraping it was a structural never-green (404), so the job was deleted - # rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26). - # The two Dankclaw endpoints are HOST processes reached through - # host.containers.internal: this Prometheus is a bridge-networked - # container, and on this netavark host a container can only reach the - # host's LAN IP at a PUBLISHED port — a host-process bind like - # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it - # answers at 169.254.1.2:7770 (why frank-zealot stayed red by construction - # for days until the 2026-09-20 /targets audit). +@test "prometheus scrapes every LLM endpoint, in its real metric namespace" { + # Every LLM endpoint that exports /metrics must have a scrape job. As of + # 2026-10-05 there are TWO runtime namespaces scraped: + # * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, + # re-exporting the llama.cpp names), both root host processes on + # Dankclaw reached via host.containers.internal — this Prometheus is a + # bridge container and a host-process bind is refused at the LAN IP, + # which is why frank-zealot read red by construction for days. + # * vllm:* — frank-archon (radiance) on Dankserver, a different machine, + # so plain LAN routing reaches 192.168.0.235:7770. It RE-JOINED the + # scrape set with radiance; the old "job deleted, exports no /metrics" + # note belonged to the retired FreeToken engine and is false today, so + # this seam now asserts archon IS scraped (a bare `! grep` here was + # set -e-exempt and never enforced the opposite). local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml" grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-zealot not scraped"; return 1; } grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-dragoon not scraped"; return 1; } - ! grep -q '"192.168.0.235:7770"' "$cfg" - grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \ - && { echo "frank-archon folded into a Dankclaw gateway target"; return 1; } - for job in frank-zealot frank-dragoon; do + grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-archon not scraped"; return 1; } + for job in frank-zealot frank-dragoon frank-archon; do grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; } done - ! grep -q 'job_name: "frank-archon"' "$cfg" - # None of them is reached through a guarded vhost: Prometheus cannot solve - # an Anubis challenge, so a franksdank.cloud target is red by construction. - ! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg" + # No LLM endpoint is reached through a guarded vhost: Prometheus cannot + # solve an Anubis challenge, so a franksdank.cloud target is red by + # construction. + grep -qE '^[[:space:]]*-[[:space:]]*"?https?://[^"]*franksdank' "$cfg" \ + && { echo "a scrape target points at a guarded vhost"; return 1; } + true } @test "grafana provisioning is declared in git and installed by the deployer" { @@ -378,28 +378,44 @@ names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rule for want in ('job:llm_prefix_cache_reuse:rate5m', 'job:llm_spec_accept_length:rate5m', 'LlmRequestsQueueing', 'LlmPrefixCacheCollapse', - 'LlmSpecDecodeOff', 'LlmEndpointDown'): + 'LlmSpecDecodeOff', 'LlmEndpointDown', + 'job:vllm_ttft_p95:rate5m', 'VllmRequestsQueueing', + 'VllmPrefixCacheCollapse', 'VllmPreempting', 'VllmTtftSlow'): assert want in names, want # Every rule must be keyed to a series that actually exists. A rule naming a # series nobody exports evaluates nothing forever and errors never — the -# failure this seam exists for. Since the 2026-09-26 cutover the only scraped -# LLM runtime is llama.cpp (frank-zealot, frank-dragoon); the vllm: family left -# with frank-medium's radiance runtime and must not survive in any expr. +# failure this seam exists for. Two runtimes are scraped in two namespaces: +# llama.cpp/halogen (llamacpp:*) for frank-zealot and frank-dragoon, and +# radiance (vllm:*) for frank-archon. Both families were read straight off the +# endpoints' /metrics; anything else is a guess. exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']] -# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20 -# (frank-zealot, frank-dragoon). Anything else is a guess. -live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total', - 'llamacpp:requests_deferred', 'llamacpp:requests_processing', - 'llamacpp:spec_decode_num_drafts_total', - 'llamacpp:spec_decode_num_accepted_tokens_total', - 'llamacpp:spec_decode_num_draft_tokens_total', - 'llamacpp:tokens_predicted_total', - 'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds', - 'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'} +live = { + # llama.cpp / halogen, measured from /metrics 2026-09-16 and 2026-09-20. + 'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total', + 'llamacpp:requests_deferred', 'llamacpp:requests_processing', + 'llamacpp:spec_decode_num_drafts_total', + 'llamacpp:spec_decode_num_accepted_tokens_total', + 'llamacpp:spec_decode_num_draft_tokens_total', + 'llamacpp:tokens_predicted_total', + 'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds', + 'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode', + # radiance (frank-archon), measured from /metrics 2026-10-20. + 'vllm:num_requests_waiting', 'vllm:gpu_prefix_cache_hit_rate', + 'vllm:num_preemptions_total', 'vllm:prompt_tokens_total', + 'vllm:time_to_first_token_seconds_bucket', +} used = {m for e in exprs for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)} assert used <= live, used - live -assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped' +# The two namespaces must not bleed across groups: a llamacpp: selector +# evaluates nothing for the radiance job and vice versa, so each group may +# reference only its own family. +per = {g['name']: {m.split(':')[0] for r in g['rules'] + for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', r.get('expr', ''))} + for g in d['groups']} +for name, fams in per.items(): + allowed = {'llamacpp'} if name == 'llm-fleet' else {'vllm'} + assert fams <= allowed, (name, fams) print('rules ok:', len(names)) " # A rule file that nothing loads evaluates nothing, and Prometheus reports -- 2.54.0 From 2626332828838bf63935c31c8b2729291a422c65 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 19:28:20 -0700 Subject: [PATCH 2/3] feat(grafana): provision the Loki datasource Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100 so logs are readable through Grafana Explore. Loki ships no web UI of its own and the guarded vhost serves only the API; this is the human surface. --- .../grafana/provisioning/datasources/loki.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 shared/grafana/provisioning/datasources/loki.yml diff --git a/shared/grafana/provisioning/datasources/loki.yml b/shared/grafana/provisioning/datasources/loki.yml new file mode 100644 index 0000000..3cbaa14 --- /dev/null +++ b/shared/grafana/provisioning/datasources/loki.yml @@ -0,0 +1,17 @@ +# Grafana datasource for Loki, declared in git rather than clicked into +# grafana.db — same reason as the Prometheus datasource: a UI-configured +# datasource is invisible to review, lost with the data volume, and unknown +# to the repository. Loki OSS ships no web UI of its own, so Explore through +# this datasource is the only way a human reads the log store. +# +# URL is the container name on dankclaw_default (the same address Caddy +# proxies to), not the published loopback port. +apiVersion: 1 + +datasources: + - name: Loki + uid: loki + type: loki + access: proxy + url: http://loki:3100 + editable: false -- 2.54.0 From 70dd7fbe6d2d1fa813ea24fd5e8337e82f510a83 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 19:28:20 -0700 Subject: [PATCH 3/3] docs(observability): correct current state and log-pipeline root cause AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a Loki datasource is provisioned beside Prometheus. Observability.md: current-state census refreshed (three LLM targets up, two rule groups, Loki datasource); logs still carry no data. Alloy.md: replace the 'wrong glob' theory with the proven cause - a rootless container cannot read the 0640 systemd-journal files (a rootless busybox read returns DENIED), and rootless quadlet logs land in the user journal the system-only source never reads. Closing it is a permission/architecture decision, not a config glob. --- AGENTS.md | 8 ++-- serverAppConfigs-docs/Infrastructure/Alloy.md | 38 +++++++++++++--- .../Infrastructure/Observability.md | 43 +++++++++++-------- 3 files changed, 63 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6dd291d..0c1cbaf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,9 +166,11 @@ bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under `dankclaw/observability/quadlet/` and their compose definitions were removed — one owner per container name. The stack is **configured and delivering as of -2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are -provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/ -llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are +2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet` +dashboard are provisioned from `shared/grafana/provisioning/`, and +`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet` +(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert ++ recording rules, and firing rules are delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/ alertmanager.container`, loopback :9093) → host-side Signal relay (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN diff --git a/serverAppConfigs-docs/Infrastructure/Alloy.md b/serverAppConfigs-docs/Infrastructure/Alloy.md index a5b6d05..4175a9c 100644 --- a/serverAppConfigs-docs/Infrastructure/Alloy.md +++ b/serverAppConfigs-docs/Infrastructure/Alloy.md @@ -51,15 +51,41 @@ Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`). -## Delivery state (2026-10-18): healthy, shipping nothing +## Delivery state (2026-10-20): healthy, shipping nothing -The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18: +The unit is `active`, the container reports `healthy`, the log says `Alloy is +ready`, and the pipeline carries no data at all. -- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18). -- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`. -- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]]. +**Root cause, proven: a rootless container cannot read the host journal.** The +journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`). +`/var/log` is bind-mounted so the files are *visible*, but a rootless container +spawns with no supplementary groups, so it is denied. Proven directly: +`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat +/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via +`sd_journal_open`, so it silently sees an empty view — no error logged, counter +frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and +`user-1000.journal` are both present and non-trivial, 2026-10-20). -No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode. +**Second, independent gap:** the rootless Quadlet containers log to the *user* +journal (`user-1000.journal`), and `loki.source.journal` reads the *system* +journal only — a fixed system read would still miss them. Container stdout is +not in a `ctr.log` here either: the file globs point at root storage +(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that +rootless does not use, refused at start with `stat …: permission denied` / `no +such file or directory`, `loki_source_file_files_active_total 0`. + +Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) — +see [[Loki]]. + +**Closing it is a permission/architecture decision, not a config edit.** The +unit, config and push endpoint are correct as written; rootless Podman simply +cannot grant the journal group. The viable shapes: run Alloy as a *system* +service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant +the container's mapped uid journal read via a tmpfiles/ACL rule; or have +journald forward to a path the container can read. Each trades rootless posture +for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total` +and the Loki label query to confirm a fix, never `/healthy`: health is exactly +what hides this failure. ## Gotchas diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index b8f3bea..190d379 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -3,7 +3,7 @@ type: reference title: "Observability on Dankclaw" description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured." tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] -timestamp: 2026-09-11 +timestamp: 2026-10-20 --- # Observability on Dankclaw @@ -83,27 +83,36 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy` [[Observability Provisioning]] for what is provisioned as code and how to verify it through the APIs. -## Current state (2026-10-18) +## Current state (2026-10-20) * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports - eight up targets; the four downs are structural (the `comfyui` and `services` - HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: - 4 alert rules + 2 recording rules (`/api/v1/rules`, matching - `shared/prometheus/rules/llm-fleet.yml`). + eight up targets — the three LLM endpoints (frank-archon on `vllm:*`, + frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus + and both node exporters. The four downs are structural (the `comfyui` and + `services` HTML probes, and Gitea's disabled metrics endpoint), not outages. + Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet` + (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4 + alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95 + time-to-first-token), each keyed only to the namespace its runtime exports. * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the relay, and `journalctl --user -u signal-relay` records successful `sent to +12062476336:` lines for every rule in the set. -* **Dashboards: working.** One provisioned Prometheus datasource (uid - `prometheus`) and the `llm-fleet` dashboard, both installed from - `shared/grafana/provisioning/`. -* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both - file globs refused) and Loki therefore holds zero streams — `/opt/loki/data` - is a few hundred KB of WAL/compactor bookkeeping with **no chunk files** - (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns - no label names. - See [[Alloy]] → Delivery state and [[Loki]] → Contents. -* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the - structural downs have no real uptime signal), no Alertmanager vhost. +* **Dashboards: working.** Provisioned datasources — Prometheus (uid + `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all + installed from `shared/grafana/provisioning/`. The Loki datasource is how + you read logs: Loki has no web UI of its own, so Grafana Explore is the + human surface (the guarded `loki.franksdank.cloud` vhost serves only the API). +* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero + streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on + disk). The cause is source-side access, now proven, not a config bug: a + rootless container cannot read the journal files (`system.journal` is + `root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED), + and the container-log globs point at a root-storage path that does not exist + under rootless. Closing it is a permission/architecture decision, not an + edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents. +* **Not configured:** no blackbox exporter (the structural downs have no real + uptime signal) and no Alertmanager vhost. The log pipeline is provisioned + but carries no data (see *Logs* above). ## Citations -- 2.54.0