Files

T

Frank 33eae71d33 docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

2026-10-05 21:19:15 -07:00

4.4 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability Provisioning

The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.

The config triad

Repo path	What it holds
`shared/prometheus/prometheus.yml`	scrape jobs (global + per-host)
`shared/prometheus/rules/llm-fleet.yml`	alert rules + recording rules
`shared/grafana/provisioning/`	datasources + dashboards (provisioned tree)

Targets and datasources reference container names (loki:3100, alloy:12345), never loopback-published ports; the LLM endpoints are reached by LAN IP (192.168.0.143:7770, :7771, 192.168.0.235:7770) because they are host-netns services, not containers. Prometheus 3 refuses HTML responses as metrics — HTML-scraped "probes" are structurally inert and need a blackbox exporter (see Stale Signal Audit).

The deploy path

scripts/deploy-remote.sh carries two copier functions:

copy_prometheus_config → /opt/prometheus/config/ on the target
copy_grafana_provisioning → /opt/grafana/provisioning/

scripts/deploy-plan.py maps changes under shared/prometheus/** and shared/grafana/** to the prometheus/grafana services, so deploy.sh dankclaw --service grafana (etc.) restarts only what the change touched. Seam tests in tests/observability_quadlet.bats pin: unit files exist, the copier functions exist, and the provisioned paths match the units' mounts.

Entry points

Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the vantage first: container name if the exporter is a container on the same netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module if it is an HTTP probe, not a metrics endpoint.

Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml. Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet. Rules reload with the prometheus service.

Add a datasource — drop a YAML file into shared/grafana/provisioning/datasources/. Point it at the container name of the backend (http://prometheus:9090), not loopback.

Add a dashboard — drop the JSON into shared/grafana/provisioning/dashboards/. Provisioned dashboards live in unified storage, not (only) the SQLite table — grep the API, not the DB, when verifying (see the verify section of Fleet Audit and Posture).

Verify through the APIs, not the filesystem

# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'

Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).

Current delivered state (2026-09-17)

Prometheus → Alertmanager (dankclaw/observability/quadlet/alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the LAN IP, the one bind a container can reach via host.containers.internal) → signal-cli. Some targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is the blackbox exporter. See Observability.

Observability — stack topology and the alert delivery chain
Stale Signal Audit — classifying down targets before "fixing" them
Quadlet Lifecycle — the units that host these services
Deploy State Triage — how the planner selects these services

4.4 KiB Raw Blame History