prometheus: drop the frank-primary scrape job (FreeToken exports no /metrics)

FreeToken has no /metrics endpoint — the job scraped 404s forever, a
structural never-green, not an outage (the Lucebox DS4.1 precedent). The
LlmEndpointDown alert regex self-adjusts (frank-.* matches existing series
only). Comments updated; seam now asserts frank-primary's absence. If
FreeToken grows /metrics, re-add the job.
This commit is contained in:
Frank
2026-10-05 19:38:14 -07:00
parent e2308dff71
commit 8618ec6dbe
3 changed files with 25 additions and 30 deletions
+6 -10
View File
@@ -98,10 +98,12 @@ scrape_configs:
# service down. Same host, same rule, so both local endpoints use the
# gateway.
#
# (2026-09-26) frank-primary is llama.cpp again (the unsloth b11030-mix
# prebuilt serving Qwen3.8-Flash-Next Q8_0) on Dankserver, a DIFFERENT
# machine: the netavark published-port quirk above is same-host only, so
# plain LAN routing reaches 192.168.0.235:7770 and the job returns.
# (2026-10-05) frank-primary left this file: FreeToken (the PR587 ROCm
# engine now serving it) has NO /metrics endpoint — the job scraped 404s
# forever, which is a structural never-green, not an outage. The
# netavark published-port quirk above is same-host only; plain LAN
# routing REACHES 192.168.0.235:7770 (health answers) — only the metrics
# path is absent. If FreeToken grows a /metrics, re-add the job here.
- job_name: "frank-small"
metrics_path: "/metrics"
static_configs:
@@ -116,9 +118,3 @@ scrape_configs:
labels:
service: frank-large
- job_name: "frank-primary"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.235:7770"]
labels:
service: frank-primary
+10 -8
View File
@@ -1,10 +1,12 @@
# Alerting and recording rules for the LLM fleet.
#
# One metric namespace: llamacpp:* — frank-small, frank-large and
# frank-primary, all llama.cpp with /metrics (the family verified from
# /metrics on 2026-09-16 and 2026-09-20). frank-primary is scraped across
# the LAN at 192.168.0.235:7770 (the netavark published-port quirk that
# forces the gateway for the Dankclaw endpoints is same-host only).
# One metric namespace: llamacpp:* — frank-small and frank-large, both
# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
# and 2026-09-20). frank-primary LEFT the family 2026-10-05: FreeToken (the
# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
# deleted rather than left never-green.
# (The netavark published-port quirk that forces the Dankclaw endpoints
# onto the gateway is same-host only.)
#
# Every expression is keyed to a series that EXISTS. Nothing matches on
# absence: when an endpoint's runtime leaves the fleet, its selectors are
@@ -110,6 +112,6 @@ groups:
description: >-
Prometheus cannot reach the LLM endpoint. frank-small and
frank-large are root units on Dankclaw: `systemctl is-active
frank-small` / `frank-large`. frank-primary is a root unit on
Dankserver: `systemctl is-active frank-primary`. All three
scraped endpoints export the llamacpp:* family.
frank-small` / `frank-large`. Both scraped endpoints export
the llamacpp:* family. (frank-primary left the scrape set
2026-10-05 — FreeToken exports no /metrics.)