command-center/agent/managed-skills/fleet-service-runtime-census/SKILL.md

name, description

name	description
fleet-service-runtime-census	Answer "what is service running right now?" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins.

Fleet service runtime census

Answer the question "what is <service> running currently?" with evidence, not from docs. Docs say what should run; the host says what does. Run all five probes, then report one table.

1. Probe the published port (works from any LAN box, no ssh)

curl -s -m 10 http://<host-ip>:<port>/health; echo
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices   # per-endpoint extras: /v1/models, /metrics

The extra route matters more than /health for identity claims:

• llama.cpp and radiance echo the requested model name — a 200 proves nothing; read /v1/models.
• halogen 404s /completion, /ping, /props, /tokenize.
• The frank-tts façade answers /health without dialing the engine, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.

2. Units + containers on the host (ssh as the deploy user, no sudo)

ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
  | grep -E "^(●|  Active|  Loaded)"; \
  podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
  podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'

• active (running) since … + Up N hours (healthy) is the answer; activating (auto-restart) with a ~50 ms exit is a preflight refusal, not an engine fault.
• NEVER prefix these podman calls with sudo. Containers live in the deploy user's rootless storage; sudo podman inspect frank-tts-engine returns no such object while the plain form succeeds. Treating that as "container missing" is a false finding.

3. Drop-ins — the override that no health gate sees

ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
            systemctl --user show <engine>.service -p DropInPaths'

DropInPaths= must be empty. A leftover bench drop-in (Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env) makes the next deploy boot a different model at a different shape while /health still answers 200 and the e2e gate passes.

4. Live env vs repo env — the only truth test

# repo side (local checkout)
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
# host side
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'

Diff them. Per repo law the env file beside the serve script is the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check git log --oneline -3 -- scripts/<svc>.env to confirm the host isn't ahead of git.

5. Report

Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.

Traps that produce false findings

• Digest-pinned images: the local image ID's creation date answers "is this stale" better than the tag; latest + Pull=never is a locally built image, not a registry pull.
• ssh user context changes what podman sees (root vs rootless) — see step 2.
• Two ports on one host are two providers in models.yml; a port mismatch is a rename or drift, not a config typo.