Frank
33eae71d33
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
96 lines
4.4 KiB
Markdown
96 lines
4.4 KiB
Markdown
---
|
|
type: reference
|
|
title: Observability Provisioning
|
|
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
|
|
tags: [observability, prometheus, grafana, provisioning, deployment]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Observability Provisioning
|
|
|
|
The monitoring stack is **provisioned as code**: dashboards, datasources,
|
|
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
|
|
No hand-edits inside containers — they are wiped on recreate and never
|
|
committed.
|
|
|
|
## The config triad
|
|
|
|
| Repo path | What it holds |
|
|
|---|---|
|
|
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
|
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
|
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
|
|
|
Targets and datasources reference **container names** (`loki:3100`,
|
|
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
|
|
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
|
|
are host-netns services, not containers. Prometheus 3 refuses HTML responses
|
|
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
|
|
exporter (see [[Stale Signal Audit]]).
|
|
|
|
## The deploy path
|
|
|
|
`scripts/deploy-remote.sh` carries two copier functions:
|
|
|
|
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
|
|
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
|
|
|
|
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
|
|
`shared/grafana/**` to the `prometheus`/`grafana` services, so
|
|
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
|
|
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
|
|
exist, the copier functions exist, and the provisioned paths match the units'
|
|
mounts.
|
|
|
|
## Entry points
|
|
|
|
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
|
vantage first: container name if the exporter is a container on the same
|
|
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
|
|
if it is an HTTP probe, not a metrics endpoint.
|
|
|
|
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
|
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
|
|
Rules reload with the prometheus service.
|
|
|
|
**Add a datasource** — drop a YAML file into
|
|
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
|
the backend (`http://prometheus:9090`), not loopback.
|
|
|
|
**Add a dashboard** — drop the JSON into
|
|
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
|
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
|
verifying (see the verify section of [[Fleet Audit and Posture]]).
|
|
|
|
## Verify through the APIs, not the filesystem
|
|
|
|
```bash
|
|
# prometheus: rules and targets actually loaded
|
|
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
|
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
|
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
|
# grafana: datasource + dashboard provisioned
|
|
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
|
```
|
|
|
|
Health answers from a stale process and provisioned resources can hide in
|
|
unified storage — a file existing on disk is not proof the running server
|
|
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
|
table (refused = outage, 404/HTML = config defect).
|
|
|
|
## Current delivered state (2026-09-17)
|
|
|
|
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
|
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
|
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
|
via `host.containers.internal`) → signal-cli. Some targets read down for
|
|
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
|
the blackbox exporter. See [[Observability]].
|
|
|
|
## Related
|
|
|
|
- [[Observability]] — stack topology and the alert delivery chain
|
|
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
|
- [[Quadlet Lifecycle]] — the units that host these services
|
|
- [[Deploy State Triage]] — how the planner selects these services
|