AGENTS.md

@@ -166,9 +166,11 @@ bare compose container never returned; 2026-09-11: five observability
 services had no units). The observability services now have units under
 `dankclaw/observability/quadlet/` and their compose definitions were removed —
 one owner per container name. The stack is **configured and delivering as of
-2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
+2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
-provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
+dashboard are provisioned from `shared/grafana/provisioning/`, and
-llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
+`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
+(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
++ recording rules, and firing rules are
 delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
 alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN

serverAppConfigs-docs/Infrastructure/Alloy.md

@@ -51,15 +51,41 @@ Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs
 
 Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
 
-## Delivery state (2026-10-18): healthy, shipping nothing
+## Delivery state (2026-10-20): healthy, shipping nothing
 
-The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
+The unit is `active`, the container reports `healthy`, the log says `Alloy is
+ready`, and the pipeline carries no data at all.
 
-- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
+**Root cause, proven: a rootless container cannot read the host journal.** The
-- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
+journal files are `root:systemd-journal 0640` (with `group:adm`, `other::---`).
-- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
+`/var/log` is bind-mounted so the files are *visible*, but a rootless container
+spawns with no supplementary groups, so it is denied. Proven directly:
+`podman run --rm -v /var/log/journal:/var/log/journal:ro busybox cat
+/var/log/journal/*/system.journal` → `DENIED`. `loki.source.journal` opens via
+`sd_journal_open`, so it silently sees an empty view — no error logged, counter
+frozen at `loki_source_journal_target_lines_total 0` (`system.journal` and
+`user-1000.journal` are both present and non-trivial, 2026-10-20).
 
-No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
+**Second, independent gap:** the rootless Quadlet containers log to the *user*
+journal (`user-1000.journal`), and `loki.source.journal` reads the *system*
+journal only — a fixed system read would still miss them. Container stdout is
+not in a `ctr.log` here either: the file globs point at root storage
+(`/var/lib/containers/storage/.../ctr.log`, `/var/log/containers/*.log`) that
+rootless does not use, refused at start with `stat …: permission denied` / `no
+such file or directory`, `loki_source_file_files_active_total 0`.
+
+Downstream: Loki holds zero streams (`GET /loki/api/v1/labels` → no `data`) —
+see [[Loki]].
+
+**Closing it is a permission/architecture decision, not a config edit.** The
+unit, config and push endpoint are correct as written; rootless Podman simply
+cannot grant the journal group. The viable shapes: run Alloy as a *system*
+service (root, in `adm`/`systemd-journal`) instead of a rootless Quadlet; grant
+the container's mapped uid journal read via a tmpfiles/ACL rule; or have
+journald forward to a path the container can read. Each trades rootless posture
+for reach — decide deliberately. Re-check `loki_source_journal_target_lines_total`
+and the Loki label query to confirm a fix, never `/healthy`: health is exactly
+what hides this failure.
 
 ## Gotchas
 

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -3,7 +3,7 @@ type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---
 
 # Observability on Dankclaw
@@ -83,27 +83,36 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.
 
-## Current state (2026-10-18)
+## Current state (2026-10-20)
 
 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets; the four downs are structural (the `comfyui` and `services`
+  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
-  HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
-  4 alert rules + 2 recording rules (`/api/v1/rules`, matching
+  and both node exporters. The four downs are structural (the `comfyui` and
-  `shared/prometheus/rules/llm-fleet.yml`).
+  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
+  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
+  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
+  time-to-first-token), each keyed only to the namespace its runtime exports.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
   relay, and `journalctl --user -u signal-relay` records successful
   `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** One provisioned Prometheus datasource (uid
+* **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and the `llm-fleet` dashboard, both installed from
+  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
-  `shared/grafana/provisioning/`.
+  installed from `shared/grafana/provisioning/`. The Loki datasource is how
-* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
+  you read logs: Loki has no web UI of its own, so Grafana Explore is the
-  file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
-  is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
+* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero
-  (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
+  streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on
-  no label names.
+  disk). The cause is source-side access, now proven, not a config bug: a
-  See [[Alloy]] → Delivery state and [[Loki]] → Contents.
+  rootless container cannot read the journal files (`system.journal` is
-* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
+  `root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED),
-  structural downs have no real uptime signal), no Alertmanager vhost.
+  and the container-log globs point at a root-storage path that does not exist
+  under rootless. Closing it is a permission/architecture decision, not an
+  edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents.
+* **Not configured:** no blackbox exporter (the structural downs have no real
+  uptime signal) and no Alertmanager vhost. The log pipeline is provisioned
+  but carries no data (see *Logs* above).
 
 ## Citations
 

shared/grafana/provisioning/datasources/loki.yml

@@ -0,0 +1,17 @@
+# Grafana datasource for Loki, declared in git rather than clicked into
+# grafana.db — same reason as the Prometheus datasource: a UI-configured
+# datasource is invisible to review, lost with the data volume, and unknown
+# to the repository. Loki OSS ships no web UI of its own, so Explore through
+# this datasource is the only way a human reads the log store.
+#
+# URL is the container name on dankclaw_default (the same address Caddy
+# proxies to), not the published loopback port.
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    uid: loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: false

shared/prometheus/rules/llm-fleet.yml

@@ -1,10 +1,15 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both
+# Two metric namespaces, because two runtime families are scraped:
-# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
+#
-# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the
+#   * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
-# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
+#     re-exports the llama.cpp metric names). Verified from /metrics on
-# deleted rather than left never-green.
+#     2026-09-16 / 2026-09-20.
+#   * vllm:* — frank-archon, the radiance engine on Dankserver since
+#     2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
+#     it is a separate scrape job in prometheus.yml and the rules below are a
+#     separate group. (The old claim that its scrape job was deleted is false:
+#     it re-joined the day it left the FreeToken engine and is `up` today.)
 # (The netavark published-port quirk that forces the Dankclaw endpoints
 # onto the gateway is same-host only.)
 #
@@ -22,7 +27,8 @@
 #     cached by cached + prompt.
 #   * Speculative decoding: drafts and accepted tokens are both counters,
 #     so LlmSpecDecodeOff watches the pair.
-#   * No latency histograms exist, so there is no p95 alert.
+#   * No latency histograms in llama.cpp, so no p95 alert there; radiance's
+#     vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
 #
 # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
 # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -112,6 +118,84 @@ groups:
           description: >-
             Prometheus cannot reach the LLM endpoint. frank-zealot and
             frank-dragoon are root units on Dankclaw: `systemctl is-active
-            frank-zealot` / `frank-dragoon`. Both scraped endpoints export
+            frank-zealot` / `frank-dragoon` (both export llamacpp:* —
-            the llamacpp:* family. (frank-archon left the scrape set
+            dragoon via halogen's llama.cpp-compatible names). frank-archon
-            2026-10-05 — FreeToken exports no /metrics.)
+            is a radiance endpoint on Dankserver (`systemctl show
+            frank-archon` there) exporting the vllm:* family; the
+            `frank-.*` selector covers all three regardless of namespace.
+
+  # frank-archon (radiance) exports the vLLM-compatible namespace, NOT
+  # llamacpp:*. Every rule here is keyed to a series the exporter actually
+  # emits (verified from /metrics 2026-10-20: num_requests_waiting,
+  # gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
+  # time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
+  # this job and these evaluate nothing for llama.cpp/halogen — two clean
+  # namespaces, each keyed only to what its runtime exports.
+  - name: vllm-fleet
+    interval: 30s
+    rules:
+      # p95 time-to-first-token, shared with the dashboard. When the endpoint
+      # is idle every bucket rate is zero, histogram_quantile yields NaN, and
+      # neither this nor the alert below can fire on an idle engine.
+      - record: job:vllm_ttft_p95:rate5m
+        expr: |
+          histogram_quantile(0.95, sum by (job, le)
+              (rate(vllm:time_to_first_token_seconds_bucket[5m])))
+
+      # Radiance exports a real queue-depth gauge (llama.cpp only shows the
+      # wall as requests_deferred). --max-num-seqs is small, so a sustained
+      # wait is either real load or this endpoint carrying the fleet alone.
+      - alert: VllmRequestsQueueing
+        expr: vllm:num_requests_waiting > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: requests waiting for 10 minutes"
+          description: >-
+            The radiance scheduler has requests queued. Either the fleet is
+            genuinely busy or this endpoint is carrying it alone.
+
+      # Radiance reports prefix reuse directly as a hit-rate gauge — no
+      # counter division needed. The token-rate floor keeps an idle engine
+      # (hit rate 0 with no traffic) from firing.
+      - alert: VllmPrefixCacheCollapse
+        expr: >-
+          vllm:gpu_prefix_cache_hit_rate < 0.20
+            and rate(vllm:prompt_tokens_total[5m]) > 0.05
+        for: 15m
+        labels:
+          severity: info
+        annotations:
+          summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
+          description: >-
+            Nearly every request prefilling from scratch. With several large
+            contexts in flight this is the capacity wall, not a routing fault.
+
+      # Preemption is the KV-thrash signature radiance reports that llama.cpp
+      # cannot: the engine evicts a running request to admit another. Any
+      # sustained rate means the KV cache cannot hold the concurrency in flight.
+      - alert: VllmPreempting
+        expr: rate(vllm:num_preemptions_total[5m]) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: engine preempting requests"
+          description: >-
+            The engine is evicting running requests to admit new ones — the KV
+            cache is undersized for the concurrency in flight.
+
+      # p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
+      # legitimately spends many seconds before token 1, so this fires only on
+      # a wedged scheduler or a queue backed up behind one request.
+      - alert: VllmTtftSlow
+        expr: job:vllm_ttft_p95:rate5m > 120
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
+          description: >-
+            First-token latency is pathological, not an honest big prefill.
+            Likely a wedged scheduler or one request stalling the queue.

tests/observability_quadlet.bats

@@ -313,33 +313,33 @@ PAIRS
     done
 }
 
-@test "prometheus scrapes every LLM endpoint that exports metrics" {
+@test "prometheus scrapes every LLM endpoint, in its real metric namespace" {
-    # The LLM endpoints are the services whose behaviour this fleet argues about
+    # Every LLM endpoint that exports /metrics must have a scrape job. As of
-    # most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw
+    # 2026-10-05 there are TWO runtime namespaces scraped:
-    # endpoints export the llamacpp:* family. frank-archon LEFT the set
+    #   * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen,
-    # 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint —
+    #     re-exporting the llama.cpp names), both root host processes on
-    # scraping it was a structural never-green (404), so the job was deleted
+    #     Dankclaw reached via host.containers.internal — this Prometheus is a
-    # rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26).
+    #     bridge container and a host-process bind is refused at the LAN IP,
-    # The two Dankclaw endpoints are HOST processes reached through
+    #     which is why frank-zealot read red by construction for days.
-    # host.containers.internal: this Prometheus is a bridge-networked
+    #   * vllm:* — frank-archon (radiance) on Dankserver, a different machine,
-    # container, and on this netavark host a container can only reach the
+    #     so plain LAN routing reaches 192.168.0.235:7770. It RE-JOINED the
-    # host's LAN IP at a PUBLISHED port — a host-process bind like
+    #     scrape set with radiance; the old "job deleted, exports no /metrics"
-    # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
+    #     note belonged to the retired FreeToken engine and is false today, so
-    # answers at 169.254.1.2:7770 (why frank-zealot stayed red by construction
+    #     this seam now asserts archon IS scraped (a bare `! grep` here was
-    # for days until the 2026-09-20 /targets audit).
+    #     set -e-exempt and never enforced the opposite).
     local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
     grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-zealot not scraped"; return 1; }
     grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-dragoon not scraped"; return 1; }
-    ! grep -q '"192.168.0.235:7770"' "$cfg"
+    grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-archon not scraped"; return 1; }
-    grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \
+    for job in frank-zealot frank-dragoon frank-archon; do
-        && { echo "frank-archon folded into a Dankclaw gateway target"; return 1; }
-    for job in frank-zealot frank-dragoon; do
         grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
     done
-    ! grep -q 'job_name: "frank-archon"' "$cfg"
+    # No LLM endpoint is reached through a guarded vhost: Prometheus cannot
-    # None of them is reached through a guarded vhost: Prometheus cannot solve
+    # solve an Anubis challenge, so a franksdank.cloud target is red by
-    # an Anubis challenge, so a franksdank.cloud target is red by construction.
+    # construction.
-    ! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
+    grep -qE '^[[:space:]]*-[[:space:]]*"?https?://[^"]*franksdank' "$cfg" \
+        && { echo "a scrape target points at a guarded vhost"; return 1; }
+    true
 }
 
 @test "grafana provisioning is declared in git and installed by the deployer" {
@@ -378,28 +378,44 @@ names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rule
 for want in ('job:llm_prefix_cache_reuse:rate5m',
              'job:llm_spec_accept_length:rate5m',
              'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
-             'LlmSpecDecodeOff', 'LlmEndpointDown'):
+             'LlmSpecDecodeOff', 'LlmEndpointDown',
+             'job:vllm_ttft_p95:rate5m', 'VllmRequestsQueueing',
+             'VllmPrefixCacheCollapse', 'VllmPreempting', 'VllmTtftSlow'):
     assert want in names, want
 # Every rule must be keyed to a series that actually exists. A rule naming a
 # series nobody exports evaluates nothing forever and errors never — the
-# failure this seam exists for. Since the 2026-09-26 cutover the only scraped
+# failure this seam exists for. Two runtimes are scraped in two namespaces:
-# LLM runtime is llama.cpp (frank-zealot, frank-dragoon); the vllm: family left
+# llama.cpp/halogen (llamacpp:*) for frank-zealot and frank-dragoon, and
-# with frank-medium's radiance runtime and must not survive in any expr.
+# radiance (vllm:*) for frank-archon. Both families were read straight off the
+# endpoints' /metrics; anything else is a guess.
 exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
-# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20
+live = {
-# (frank-zealot, frank-dragoon). Anything else is a guess.
+    # llama.cpp / halogen, measured from /metrics 2026-09-16 and 2026-09-20.
-live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
+    'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
-        'llamacpp:requests_deferred', 'llamacpp:requests_processing',
+    'llamacpp:requests_deferred', 'llamacpp:requests_processing',
-        'llamacpp:spec_decode_num_drafts_total',
+    'llamacpp:spec_decode_num_drafts_total',
-        'llamacpp:spec_decode_num_accepted_tokens_total',
+    'llamacpp:spec_decode_num_accepted_tokens_total',
-        'llamacpp:spec_decode_num_draft_tokens_total',
+    'llamacpp:spec_decode_num_draft_tokens_total',
-        'llamacpp:tokens_predicted_total',
+    'llamacpp:tokens_predicted_total',
-        'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
+    'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
-        'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'}
+    'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode',
+    # radiance (frank-archon), measured from /metrics 2026-10-20.
+    'vllm:num_requests_waiting', 'vllm:gpu_prefix_cache_hit_rate',
+    'vllm:num_preemptions_total', 'vllm:prompt_tokens_total',
+    'vllm:time_to_first_token_seconds_bucket',
+}
 used = {m for e in exprs
         for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
 assert used <= live, used - live
-assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped'
+# The two namespaces must not bleed across groups: a llamacpp: selector
+# evaluates nothing for the radiance job and vice versa, so each group may
+# reference only its own family.
+per = {g['name']: {m.split(':')[0] for r in g['rules']
+                   for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', r.get('expr', ''))}
+       for g in d['groups']}
+for name, fams in per.items():
+    allowed = {'llamacpp'} if name == 'llm-fleet' else {'vllm'}
+    assert fams <= allowed, (name, fams)
 print('rules ok:', len(names))
 "
     # A rule file that nothing loads evaluates nothing, and Prometheus reports