169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
vllm:prefix_cache_hit_rate does not exist in this build; the exported pair is
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give
the hit rate. A comment naming a series nobody can query is worse than no
comment.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
Frank