chore(llm): add dankstatus.sh — one verified parse path for Dankserver state

The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so

  grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2

returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.

dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.

Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.

Tests: 346/346 pass (no test surface touched).
This commit is contained in:
Frank
2026-09-28 23:27:50 -07:00
parent b4b6777e49
commit 32abc8d543
8 changed files with 525 additions and 146 deletions
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Single source of truth for Dankserver GLM-5.3 state.
#
# This exists because the ad-hoc status commands kept producing WRONG readings.
# The specific trap: /proc/meminfo separates key from value with a TAB, so the
# common idiom
# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
# reads as "host out of memory" when the host actually has 240 GiB free.
# Everything below parses on a colon split, which is tab-agnostic, and the
# memory figure is cross-checked before it is shown.
set -uo pipefail
MB=1048576
GLM_CTR="${GLM_CTR:-glm-w4a16}"
mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
section() { printf '\n== %s ==\n' "$1"; }
section "time / host"
printf ' %s %s\n' "$(date +%H:%M:%S)" \
"$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
section "memory (GiB)"
TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
printf ' MemTotal %8s\n' "$TOT"
printf ' MemAvailable %8s\n' "$AVAIL"
printf ' Cached %8s\n' "$(mem_field Cached)"
awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit }
if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit }
printf " check ok (%.0f%% free)\n", 100*a/t
}'
section "swap"
swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
| awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none"
section "gpu"
# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
NR>1 && $1 ~ /^card[0-9]+$/ {
printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
}'
printf ' compute util: '
rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
echo
section "disk"
df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}'
section "llm containers"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
| awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}'
else
echo " none"
fi
section "frank-primary unit"
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
section "glm serve tail"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /'
else
echo " (container $GLM_CTR not present)"
fi
section "done"