Frank 650b6e0281 fix(observability): serve every exporter bind; scrape same-host cadvisor via gateway
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:

- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
  serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
  but never accept()ed, loopback deploy gate passed, every scrape that
  reached the other bind timed out on the SYN backlog. Now one thread per
  extra bind. Regression test spins --serve on two loopback binds and
  requires BOTH to answer /metrics.

- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
  Prometheus container: a bridge container cannot dial its own host's LAN
  address at all (hairpin refuses published ports too); the podman gateway
  answers. Target moved to host.containers.internal:8080; the seam that
  pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
  gateway plus a negative pin against the own-host LAN dial. Observability
  page records the measured reachability law replacing the inferred one.
2026-10-21 00:34:17 -07:00
2026-10-15 18:22:22 -07:00

Server App Configs

AI Disclosures:

  • Slop rating: 2/5 - Actively using, actively de-slopping, repository mostly human owned, lacks formal process.
  • Completeness: Unfinished, planning planned, no plan yet
  • Datacenter model/spend: Less than $20 of Deepseek 4.1 Flash and GLM-5.3-Flash
  • Local model: Qwen 3.8 Flash Next

slop rating of 2 or lower indicates human reading is a worthwhile exercise, and that Frank has read the code and knows the architecture.

slop rating of 2 indicates that while this project is human owned, no human contribution process exists and/or release is agentically driven.

The single repository for the two homelab servers: the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script.

2026-08-22: the former vllmScripts repository (LLM fleet) is merged into this repo with its full git history. See AGENTS.md for the merge note.

Targets

Host User IP Podman stack LLM service LLM port
dankserver dank 192.168.0.235 media and library frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) 7770
dankclaw dank 192.168.0.143 AI, search, development, observability frank-dragoon (gufo, Qwen3.8-27B UD-Q8_K_XL) 7770

Active Services

Service Host Hostname Port
RomM dankserver romm.dankserver.local 8085
Copyparty dankserver files.dankserver.local 3923
Calibre-Web dankserver books.dankserver.local 8083
frank-medium dankserver — (bare endpoint) 7770
SearXNG dankclaw search.dankclaw.local 8081
ComfyUI dankclaw comfyui.dankclaw.local 8188
OpenWebUI dankclaw chat.dankclaw.local 3000
Hermes dankclaw hermes.dankclaw.local 8090
Gitea dankclaw gitea.dankclaw.local 3001
Grafana dankclaw grafana.dankclaw.local 3002
Prometheus dankclaw prometheus.dankclaw.local 9090
frank-dragoon dankclaw — (bare endpoint) 7770

Structure

  • dankserver/ — compose.yaml + Caddyfile + .env.example for the Dankserver stack
  • dankclaw/ — compose.yaml + Caddyfile + .env.example for the Dankclaw stack
  • shared/ — homepage, prometheus config, deploy helpers
  • scripts/ — deploy.sh and deploy-remote.sh (service-scoped deployment), the planner, LLM serve scripts, systemd units, env tunables, and bench harnesses
  • old/ — retired PP/NPU-era LLM scripts (reference only)
  • serverAppConfigs-docs/ — project knowledge base (Obsidian vault, wikilinks). serverAppConfigs-docs/LLM/ is the LLM fleet
  • tests/ — bats test suite (see Testing Policy)

Deployment

See serverAppConfigs-docs/Architecture/Deployment Flow.md and serverAppConfigs-docs/LLM/deployment.md. Run ./scripts/deploy.sh <dankclaw|dankserver> after committing changes. It plans and deploys only the changed services. Use --service NAME for an explicit single-service retry; inference is restarted only when its own files (or the shared chat template) changed.

Knowledge Base

The serverAppConfigs-docs/ directory is an Obsidian vault. Start at serverAppConfigs-docs/index.md. The LLM fleet knowledge base lives at serverAppConfigs-docs/LLM/index.md.

Testing

Run ./tests/run.sh before every push. See serverAppConfigs-docs/Architecture/Testing Policy.md.

S
Description
Podman deployment configs for Dankserver and Dankclaw
Readme
9.1 MiB
Languages
Shell 60.6%
Python 20.9%
HTML 6%
Jinja 3.8%
C++ 3.8%
Other 4.9%