Frank
1919ad7f31
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8 payload at native block-FP8 precision: - scripts/build-frank-primary-rad.sh: staging (quantization_config drop = plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE --quant rule (PLE rtn fixed-scale); plan-only first. - scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes (experts byte-stacked into the plugin's stacked layout; PLE's 128 scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own scale; the rtn rule re-encodes at that same scale - codes round-trip exactly and land the one-element scale plane the plugin declares). - scripts/hfget.sh + verified download (131/131 shards sha256-match). - froggeric v22.5 chat template vendored (--override-chat-template). - serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host), TTM preflight fixed for kernels with pages_limit only (no pages_max - an absent knob is skipped, not read as zero); pages_limit raised to 52,428,800 pages = 200 GiB on the host pre-cutover. - root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB + health; prometheus re-adds the /metrics job. - rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile); payload + image stay on disk until the window closes. deploy-remote.sh also carries the parallel session's frank-small payload gates (swift15) - committed here per Frank's direction to commit the tree.
134 lines
5.3 KiB
YAML
134 lines
5.3 KiB
YAML
# Prometheus configuration for the homelab.
|
|
# Deployed to /opt/prometheus/config/prometheus.yml on Dankclaw.
|
|
|
|
global:
|
|
scrape_interval: 15s
|
|
scrape_timeout: 10s
|
|
evaluation_interval: 15s
|
|
|
|
# Alerting and recording rules for the LLM fleet. They evaluate, expose
|
|
# state through /alerts and the ALERTS metric, and — since 2026-09-17 —
|
|
# deliver to Signal through Alertmanager (see the alerting section below).
|
|
rule_files:
|
|
- /etc/prometheus/rules/*.yml
|
|
|
|
# The delivery half. Alertmanager runs on the same dankclaw_default network,
|
|
# so it is addressed by container name, like grafana and loki below. Its
|
|
# single receiver webhooks the host-side signal-relay (192.168.0.143:18093,
|
|
# reached from the container as host.containers.internal).
|
|
alerting:
|
|
alertmanagers:
|
|
- static_configs:
|
|
- targets: ["alertmanager:9093"]
|
|
|
|
scrape_configs:
|
|
# Prometheus itself
|
|
- job_name: "prometheus"
|
|
static_configs:
|
|
- targets: ["localhost:9090"]
|
|
|
|
# Host metrics — both hosts. node-dankclaw is scraped over
|
|
# host.containers.internal, not its own LAN address: this Prometheus is a
|
|
# bridge container, and on this netavark host a container can only reach the
|
|
# host's LAN IP at a PUBLISHED port — node-exporter's net=host bind on
|
|
# 192.168.0.143:9100 is refused by construction (the same rule that sends
|
|
# frank-large and frank-small to the gateway below). node-dankserver is a
|
|
# different machine, so its LAN address works.
|
|
- job_name: "node-dankclaw"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:9100"]
|
|
|
|
- job_name: "node-dankserver"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:9100"]
|
|
|
|
# Gitea metrics (quadlet on an isolated network; use the published LAN port)
|
|
- job_name: "gitea"
|
|
static_configs:
|
|
- targets: ["192.168.0.143:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Grafana metrics
|
|
- job_name: "grafana"
|
|
static_configs:
|
|
- targets: ["grafana:3000"]
|
|
metrics_path: "/metrics"
|
|
|
|
# Loki metrics
|
|
- job_name: "loki"
|
|
static_configs:
|
|
- targets: ["loki:3100"]
|
|
metrics_path: "/metrics"
|
|
|
|
# ComfyUI (no native /metrics; probe the UI)
|
|
- job_name: "comfyui"
|
|
metrics_path: "/"
|
|
static_configs:
|
|
- targets: ["comfyui:8188"]
|
|
|
|
# Blackbox-style probes for service up/down (published LAN ports;
|
|
# gitea and comfyui are covered by their own jobs above)
|
|
- job_name: "services"
|
|
metrics_path: "/"
|
|
static_configs:
|
|
- targets:
|
|
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
|
- "192.168.0.143:3005" # OpenWebUI
|
|
|
|
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
|
|
# this container, not the guarded vhosts: these are machine clients and
|
|
# Prometheus cannot solve an Anubis challenge.
|
|
#
|
|
# One metric namespace: llamacpp:* — frank-large, frank-small and
|
|
# frank-primary, all llama.cpp with /metrics (the family verified from
|
|
# /metrics on 2026-09-16 and 2026-09-20) and all exporting
|
|
# requests_deferred (the queue depth, the only form llama.cpp shows the
|
|
# capacity wall in), spec_decode_num_drafts_total, tokens_predicted_total,
|
|
# and the prompt_tokens_cached_total / prompt_tokens_total reuse pair.
|
|
#
|
|
# frank-large and frank-small are scraped over host.containers.internal,
|
|
# not their own LAN addresses: this Prometheus is a bridge-networked
|
|
# container (Network=dankclaw_default), and on this netavark host a
|
|
# container can only reach the host's LAN IP at a PUBLISHED port — a
|
|
# host-process bind like llama-server's 0.0.0.0:7770 is refused at
|
|
# 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
|
|
# why frank-small was red here for days (up{job="frank-small"} == 0,
|
|
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
|
|
# exporting: the target address was unreachable by construction, not the
|
|
# service down. Same host, same rule, so both local endpoints use the
|
|
# gateway.
|
|
#
|
|
# (2026-10-05) frank-primary LEFT the llamacpp family with FreeToken
|
|
# (the PR587 ROCm engine then serving it had NO /metrics — the job
|
|
# scraped 404s forever, a structural never-green) and RE-JOINED the same
|
|
# day on the radiance engine: its exporter speaks vLLM-compatible names
|
|
# plus radiance:* families, a different namespace — hence the separate
|
|
# job below, not a re-add to the llamacpp pair. The netavark quirk above
|
|
# is same-host only; plain LAN routing REACHES 192.168.0.235:7770.
|
|
- job_name: "frank-small"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:7771"]
|
|
labels:
|
|
service: frank-small
|
|
|
|
- job_name: "frank-large"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["host.containers.internal:7770"]
|
|
labels:
|
|
service: frank-large
|
|
|
|
# frank-primary (dankserver) — the radiance engine since 2026-10-05.
|
|
# A DIFFERENT host, so plain LAN routing is correct here (the
|
|
# host.containers.internal trick is same-host only): health answers at
|
|
# 192.168.0.235:7770 from this bridge container. Radiance exports
|
|
# vLLM-compatible names plus radiance:* families — NOT the llamacpp:*
|
|
# namespace, so llm-fleet rules must not be pointed at this target.
|
|
- job_name: "frank-primary"
|
|
metrics_path: "/metrics"
|
|
static_configs:
|
|
- targets: ["192.168.0.235:7770"]
|
|
labels:
|
|
service: frank-primary
|