feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank committed 2026-09-20 19:35:56 -07:00
1 parent c2116658f8
commit 08f1c7ad09
13 files changed
+687 -286

No files matched your search

+36 -19
View File
@@ -69,34 +69,51 @@ scrape_configs:
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
- "192.168.0.143:3005" # OpenWebUI
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
# machine clients and Prometheus cannot solve an Anubis challenge.
# Both LLM endpoints are llama.cpp, so both export the same llamacpp:
# counter family: prompt/decode throughput, requests_deferred (queue depth
# — the only form llama.cpp exposes the capacity wall in), the
# spec_decode_num_drafts / spec_decode_num_accepted pair (proof the drafter
# is running), and the prefix-cache pair prompt_tokens_cached_total over
# prompt_tokens_total (their rates are the reuse rate, which says whether
# pinned conversations are actually being reused).
# frank-large is deliberately NOT scraped: the halogen engine that serves it
# publishes /health, /cache and /v1/* and no Prometheus text endpoint, so a
# target here would be red by construction and would fire LlmEndpointDown
# forever. Its liveness is on the homepage (a loopback probe) and in the
# engine's own boot ledger. The remedy, when the fleet wants it, is the
# blackbox exporter already wanted for the comfyui/services probes.
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
# this container, not the guarded vhosts: these are machine clients and
# Prometheus cannot solve an Anubis challenge.
#
# frank-small is llama.cpp (its serve script passes --metrics) and does
# export text-format counters under llamacpp: on the port its unit publishes.
# Two metric namespaces, because the fleet runs two runtimes:
#
# * vllm:* — frank-medium (vllm-radiance TP=2 since 2026-09-20): prompt and
# generation token counters, num_requests_waiting plus its reason label,
# num_preemptions_total, the kv_cache_usage_perc fill gauge, the
# prefix_cache hits/queries pair, the spec_decode drafts/accepted triple,
# and real latency histograms (time_to_first_token, inter_token,
# e2e_request_latency).
# * llamacpp:* — frank-large and frank-small. Both take --metrics and both
# were verified 2026-09-20 to export requests_deferred (the queue depth,
# the only form llama.cpp shows the capacity wall in),
# spec_decode_num_drafts_total, tokens_predicted_total, and the
# prompt_tokens_cached_total / prompt_tokens_total reuse pair.
#
# frank-large is scraped over host.containers.internal, not its own LAN address:
# this Prometheus is a bridge-networked container (Network=dankclaw_default),
# and on this netavark host a container can only reach the host's LAN IP at a
# PUBLISHED port — a host-process bind like llama-server's 0.0.0.0:7770 is
# refused at 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
# why frank-small was red here for days (up{job="frank-small"} == 0,
# "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
# exporting: the target address was unreachable by construction, not the
# service down. Same host, same rule, so both local endpoints use the gateway.
# frank-medium stays a LAN address — it is a different machine.
- job_name: "frank-medium"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.235:7770"]
labels:
service: frank-medium
- job_name: "frank-small"
metrics_path: "/metrics"
static_configs:
- targets: ["192.168.0.143:7771"]
- targets: ["host.containers.internal:7771"]
labels:
service: frank-small
- job_name: "frank-large"
metrics_path: "/metrics"
static_configs:
- targets: ["host.containers.internal:7770"]
labels:
service: frank-large