docs: the vault and AGENTS.md describe two Dankclaw endpoints
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget 33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything else), ik_llama.cpp is retired with its measurements kept as the record of why, and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV arithmetic, the MTP contract, and the reasoning_effort trap that cost me every subagent on the endpoint. AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's template can reject the harness's thinking level). Three pre-existing defects found while editing it: a truncated sentence mid-file, a harness path that does not exist on either server, and the benchmarking stop/mask recipe still calling frank-large a root unit. Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the claim that it "no longer prints" is gone and the three observed block counts are recorded instead of a guessed threshold; and the Observability page says out loud why frank-large has no scrape job rather than leaving a silent absence.
This commit is contained in:
@@ -86,24 +86,34 @@ The rule is the same as for every other service: the only off-host path is a gua
|
||||
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
|
||||
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
|
||||
| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
|
||||
| `frank-large` | `192.168.0.143:7770` | llama.cpp on Dankclaw — `llamacpp:` counters |
|
||||
| `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
|
||||
|
||||
The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing
|
||||
frank-medium by hand. vLLM exports exactly what that diagnosis needed:
|
||||
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
|
||||
The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
|
||||
frank-medium by hand, and re-cut the same day when Dankclaw split into two
|
||||
endpoints. vLLM exports exactly what that diagnosis needed:
|
||||
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
|
||||
`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
|
||||
the prefix-cache counters `vllm:prefix_cache_hits_total` /
|
||||
`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
|
||||
whether a pinned conversation is being served warm or re-prefilled cold. What
|
||||
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]].
|
||||
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]];
|
||||
llama.cpp's counters (prompt and decode throughput, KV fill, and the
|
||||
`spec_decode_*` pair that proves MTP is running) are read against
|
||||
[[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
Both are scraped on their LAN addresses, never on the guarded vhosts:
|
||||
Prometheus cannot solve an Anubis challenge.
|
||||
**`frank-large` is deliberately NOT scraped.** The halogen engine publishes
|
||||
`/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a
|
||||
target there would be red by construction and would fire `LlmEndpointDown`
|
||||
forever — the same class of defect as the page-only probes below. Its liveness
|
||||
is on the homepage (a loopback probe) and in the engine's own boot ledger.
|
||||
|
||||
Both scraped endpoints are reached on their LAN addresses, never through the
|
||||
guarded vhosts: Prometheus cannot solve an Anubis challenge.
|
||||
|
||||
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
|
||||
|
||||
**Four jobs read "down" and three of them always will** (found 2026-09-16 while
|
||||
auditing why the target list was not green):
|
||||
**Four targets read "down" — three jobs, and every one of them is structural**
|
||||
(found 2026-09-16 while auditing why the target list was not green):
|
||||
|
||||
| job | error | what it means |
|
||||
|-----|-------|---------------|
|
||||
|
||||
Reference in New Issue
Block a user