docs: the vault and AGENTS.md describe two Dankclaw endpoints

halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.

AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.

Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
This commit is contained in:
Frank
2026-09-16 14:52:51 -07:00
parent 40850bbc8f
commit 49e8dfff66
8 changed files with 565 additions and 153 deletions
@@ -86,24 +86,34 @@ The rule is the same as for every other service: the only off-host path is a gua
| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
| `frank-large` | `192.168.0.143:7770` | llama.cpp on Dankclaw — `llamacpp:` counters |
| `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
The two LLM endpoints were added on 2026-09-16, after a night spent diagnosing
frank-medium by hand. vLLM exports exactly what that diagnosis needed:
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
frank-medium by hand, and re-cut the same day when Dankclaw split into two
endpoints. vLLM exports exactly what that diagnosis needed:
`vllm:kv_cache_usage_perc` (how full the per-card pool is),
`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
the prefix-cache counters `vllm:prefix_cache_hits_total` /
`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
whether a pinned conversation is being served warm or re-prefilled cold. What
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]].
those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]];
llama.cpp's counters (prompt and decode throughput, KV fill, and the
`spec_decode_*` pair that proves MTP is running) are read against
[[frank-small-qwen38-gsq-rco]].
Both are scraped on their LAN addresses, never on the guarded vhosts:
Prometheus cannot solve an Anubis challenge.
**`frank-large` is deliberately NOT scraped.** The halogen engine publishes
`/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a
target there would be red by construction and would fire `LlmEndpointDown`
forever — the same class of defect as the page-only probes below. Its liveness
is on the homepage (a loopback probe) and in the engine's own boot ledger.
Both scraped endpoints are reached on their LAN addresses, never through the
guarded vhosts: Prometheus cannot solve an Anubis challenge.
For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
**Four jobs read "down" and three of them always will** (found 2026-09-16 while
auditing why the target list was not green):
**Four targets read "down" — three jobs, and every one of them is structural**
(found 2026-09-16 while auditing why the target list was not green):
| job | error | what it means |
|-----|-------|---------------|