live: heartbeat sync 2026-10-19T03:04:41-07:00 (Refs #11)
This commit is contained in:
Frank
@@ -0,0 +1,62 @@
|
||||
---
|
||||
name: fleet-service-runtime-census
|
||||
description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins."
|
||||
---
|
||||
|
||||
# Fleet service runtime census
|
||||
|
||||
Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table.
|
||||
|
||||
## 1. Probe the published port (works from any LAN box, no ssh)
|
||||
|
||||
```bash
|
||||
curl -s -m 10 http://<host-ip>:<port>/health; echo
|
||||
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices # per-endpoint extras: /v1/models, /metrics
|
||||
```
|
||||
|
||||
The extra route matters more than `/health` for identity claims:
|
||||
- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`.
|
||||
- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`.
|
||||
- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.
|
||||
|
||||
## 2. Units + containers on the host (ssh as the deploy user, no sudo)
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
|
||||
| grep -E "^(●| Active| Loaded)"; \
|
||||
podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
|
||||
podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
|
||||
```
|
||||
|
||||
- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
|
||||
- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
|
||||
|
||||
## 3. Drop-ins — the override that no health gate sees
|
||||
|
||||
```bash
|
||||
ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
|
||||
systemctl --user show <engine>.service -p DropInPaths'
|
||||
```
|
||||
|
||||
`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
|
||||
|
||||
## 4. Live env vs repo env — the only truth test
|
||||
|
||||
```bash
|
||||
# repo side (local checkout)
|
||||
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
|
||||
# host side
|
||||
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
|
||||
```
|
||||
|
||||
Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
|
||||
|
||||
## 5. Report
|
||||
|
||||
Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
|
||||
|
||||
## Traps that produce false findings
|
||||
|
||||
- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
|
||||
- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
|
||||
- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: strix-halo-npu-readiness-census
|
||||
description: "Engine-agnostic census of whether an AMD Strix Halo / Ryzen AI host can run anything on its XDNA2 NPU: driver bind, firmware, device perms, IOMMU, distro XRT, missing AIE compiler, fabric clock, RAM headroom."
|
||||
---
|
||||
|
||||
Use when asked "can we run <engine/model> on the Strix Halo / Ryzen AI NPU?" for a runtime that is **not** halogen (halogen has its own gate + vendor catalog: use `halogen-npu-feasibility-census` for that). This answers only: is the *host* able, and what is missing.
|
||||
|
||||
## Why it is usually the wrong door first
|
||||
|
||||
XDNA2 executes **precompiled AIE partition programs** loaded through XRT → `amdxdna`. There is no generic tensor dispatch: an engine built on ggml/llama.cpp (or any GGUF loader) cannot "enable" the NPU — the port must be a re-export + vendor compile, not a flag. So pair this census with the accelerator the same engine *can* select by build flag (Vulkan/HIP on the gfx1151 iGPU) before proposing NPU work.
|
||||
|
||||
## Census (all read-only, over ssh from the operator box)
|
||||
|
||||
```bash
|
||||
ssh <host> '
|
||||
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l # iommu=pt required; groups==0 => IOMMU off => NPU dead
|
||||
lsmod | grep -i amdxdna # in-tree driver
|
||||
sudo dmesg | grep -iE "xdna|npu" | tail # "Initialized amdxdna_accel_driver <ver> for 0000:c6:00.1"
|
||||
lspci -nn | grep -iE "npu|processing" # Strix Halo = 1022:17f0
|
||||
ls /sys/bus/pci/drivers/amdxdna # must list the BDF
|
||||
ls -l /dev/accel # accel0, root:render 660
|
||||
id -nG <deploy-user> | tr " " "\n" | grep -E "render|video" # rootless podman needs render, else --device fails at create
|
||||
find /usr/lib/firmware -iname "*npu*" | head # firmware is /usr/lib/firmware/amdnpu/<devid>_<rev>/npu.sbin*.zst — match the PCI rev
|
||||
dpkg -l | grep -iE "xrt|xdna"; ls /opt/xilinx/xrt/lib 2>&1 # distro libxrt-npu2 = XDNA2 runtime; vendor /opt/xilinx/xrt usually absent
|
||||
apt-cache policy libxrt2 libxrt-npu2 | grep Candidate
|
||||
apt-cache search onnxruntime | head # distro ONNX RT exists; the Ryzen AI EP does not
|
||||
python3 -c "import onnxruntime" 2>&1 | tail -1 # absence of the EP + of ryzenai-compile is the real blocker
|
||||
for c in /sys/class/drm/card*/device; do echo "-- $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done
|
||||
nproc; free -g | head -2 # RAM headroom + thread count the engine will pick
|
||||
'
|
||||
```
|
||||
|
||||
## Reading it
|
||||
|
||||
- Driver bound + `accel0` + firmware dir + `iommu=pt` ⇒ **kernel side is ready**; do not re-litigate "is the NPU disabled". A missing device node is *not* the modern failure mode.
|
||||
- `auto` on `pp_dpm_fclk` / `power_dpm_force_performance_level` ⇒ fabric clock **unheld**. GPU+NPU concurrently with a moving fclk hangs these hosts and corrupts NPU results; any NPU plan must pin the top fclk step (or run in a container holding it) first.
|
||||
- No `ryzenai-compile`/ONNX Ryzen AI EP ⇒ nothing on the host can produce a partition program. That is the enablement delta, and it is a toolchain install + a model-specific export/quantize/compile project, not a config change.
|
||||
- `free` headroom matters: NPU partitions want a few GB of host RAM each and can log benign `Failed to alloc dev bo memory, ret -28` under pressure.
|
||||
|
||||
## Laws / traps
|
||||
|
||||
- Read state only. Never probe or load the NPU beside a live inference service — stop + mask the service first (fleet benching policy); this host class has a precedent of a concurrent GPU/NPU experiment needing a power cycle.
|
||||
- `sudo podman ...` finds nothing (per-user rootless storage); run podman as the deploy user.
|
||||
- `/dev/accel/accel0` existing does not mean a context/PASID can be created; with IOMMU off the failure is at context creation, not at the node.
|
||||
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792386375195
|
||||
"lastConsolidatedAt": 1792401025778
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792400553633,
|
||||
"lastConsolidatedAt": 1792400853634,
|
||||
"lastResult": {
|
||||
"at": 1792393886960,
|
||||
"sessions": 1,
|
||||
|
||||
@@ -1,4 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792400552453
|
||||
"lastConsolidatedAt": 1792404212471,
|
||||
"lastResult": {
|
||||
"at": 1792401456699,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792404069868,
|
||||
"lastResult": {
|
||||
"at": 1792402864677,
|
||||
"sessions": 1,
|
||||
"deltas": 8,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1792404208554,
|
||||
"lastResult": {
|
||||
"at": 1792401394555,
|
||||
"sessions": 1,
|
||||
"deltas": 1,
|
||||
"model": "frank-zealot"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user