From 3037240f4f89d4a481ce5a217e7740b83b3e89f7 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 06:37:39 -0700 Subject: [PATCH] docs: point the frank-medium runbook and observability page at the new scrape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The capacity work in this runbook was done with ad-hoc probes; the same signals now have history in Prometheus, so the page says which series to watch and admits the honest gap — this stack has no alerting, so a saturated pool shows up on a graph after the fact rather than paging anyone. --- .../Infrastructure/Observability.md | 14 ++++++++++++++ .../LLM/frank-medium-radiance-mxfp4.md | 12 ++++++++++++ 2 files changed, 26 insertions(+) diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 6500bb5..fddf906 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -76,6 +76,20 @@ The rule is the same as for every other service: the only off-host path is a gua | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` | | `comfyui` | `comfyui:8188` | UI probe, no native exporter | | `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes | +| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series | +| `frank-large` | `192.168.0.143:7770` | llama.cpp on Dankclaw — `llamacpp:` counters | + +The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing +frank-medium by hand. vLLM exports exactly what that diagnosis needed: +`vllm:kv_cache_usage_perc` (how full the per-card pool is), +`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and +the prefix-cache counters `vllm:prefix_cache_hits_total` / +`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says +whether a pinned conversation is being served warm or re-prefilled cold. What +those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]]. + +Both are scraped on their LAN addresses, never on the guarded vhosts: +Prometheus cannot solve an Anubis challenge. For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host. diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 9d1a50b..9d2d2ee 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -409,6 +409,18 @@ co-tenants evict each other turn by turn whatever the concurrency limit is. A card can hold two large families' caches, or one cache plus a live generation, but not both. That is the wall, and it is capacity, not scheduling. +### Watching it instead of measuring it by hand + +Everything above was diagnosed with ad-hoc probes. The same signals are now +scraped — Prometheus has `frank-medium` and `frank-large` jobs +(`shared/prometheus/prometheus.yml`), so the numbers that matter have history: +`vllm:kv_cache_usage_perc` (per-card pool fill), `vllm:num_requests_waiting` +(queue depth), `rate(vllm:prefix_cache_hits_total[5m]) / + rate(vllm:prefix_cache_queries_total[5m])` (whether pinned conversations are +served warm), and the request-latency histograms. See +[[Observability on Dankclaw]]. There is still **no alerting** in this stack, so +a saturated pool will not page anyone — it shows up on a graph after the fact. + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer