chore(llm): add dankstatus.sh — one verified parse path for Dankserver state
The ad-hoc status commands kept producing wrong readings, most seriously reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free. Cause: /proc/meminfo separates key from value with a TAB, so grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2 returns an empty field and awk prints 0.0. A false "host is out of memory" reading is worse than no reading, because it invites an unnecessary abort. dankstatus.sh parses on a colon split (tab-agnostic), cross-checks MemAvailable against MemTotal before printing either, and refuses to print a zero value. It also pins the rocm-smi CSV column order (device, total, used) which was previously read swapped, and takes only the first line of systemctl is-active/is-enabled, which emit more than one line here. Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU) as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing the GTT aperture. The device is smaller than documented, not larger. Tests: 346/346 pass (no test surface touched).
This commit is contained in:
Executable
+71
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env bash
|
||||
# Single source of truth for Dankserver GLM-5.3 state.
|
||||
#
|
||||
# This exists because the ad-hoc status commands kept producing WRONG readings.
|
||||
# The specific trap: /proc/meminfo separates key from value with a TAB, so the
|
||||
# common idiom
|
||||
# grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
|
||||
# yields an empty field (tab is not a space) and awk then prints 0.0 -- which
|
||||
# reads as "host out of memory" when the host actually has 240 GiB free.
|
||||
# Everything below parses on a colon split, which is tab-agnostic, and the
|
||||
# memory figure is cross-checked before it is shown.
|
||||
set -uo pipefail
|
||||
|
||||
MB=1048576
|
||||
GLM_CTR="${GLM_CTR:-glm-w4a16}"
|
||||
|
||||
mem_field() { awk -v k="$1:" '$1==k {printf "%.1f", $2/'"$MB"'; exit}' /proc/meminfo; }
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
section "time / host"
|
||||
printf ' %s %s\n' "$(date +%H:%M:%S)" \
|
||||
"$(awk -F: '/model name/{print $2; exit}' /proc/cpuinfo | sed 's/^ *//')"
|
||||
|
||||
section "memory (GiB)"
|
||||
TOT=$(mem_field MemTotal); AVAIL=$(mem_field MemAvailable)
|
||||
printf ' MemTotal %8s\n' "$TOT"
|
||||
printf ' MemAvailable %8s\n' "$AVAIL"
|
||||
printf ' Cached %8s\n' "$(mem_field Cached)"
|
||||
awk -v t="$TOT" -v a="$AVAIL" 'BEGIN{
|
||||
if (t+0 <= 0 || a+0 <= 0) { print " *** PARSE BUG: zero value ***"; exit }
|
||||
if (a+0 > t+0) { print " *** PARSE BUG: available > total ***"; exit }
|
||||
printf " check ok (%.0f%% free)\n", 100*a/t
|
||||
}'
|
||||
|
||||
section "swap"
|
||||
swapon --show=NAME,SIZE,USED --noheadings 2>/dev/null \
|
||||
| awk '{printf " %-12s %6s used of %s\n",$1,$3,$2}' || echo " none"
|
||||
|
||||
section "gpu"
|
||||
# CSV layout: device,VRAM Total Memory (B),VRAM Total Used Memory (B)
|
||||
rocm-smi --showmeminfo vram --csv 2>/dev/null | awk -F, '
|
||||
NR>1 && $1 ~ /^card[0-9]+$/ {
|
||||
printf " %-8s %7.2f GiB used of %.2f\n", $1, $3/1073741824, $2/1073741824
|
||||
}'
|
||||
printf ' compute util: '
|
||||
rocm-smi --showuse --csv 2>/dev/null | awk -F, 'NR>1 && $1 ~ /^card[0-9]+$/ {printf "%s=%s%% ", $1, $2}'
|
||||
echo
|
||||
|
||||
section "disk"
|
||||
df -h /home/dank 2>/dev/null | awk 'NR==2{printf " %s %s used %s free (%s)\n",$1,$3,$4,$5}'
|
||||
|
||||
section "llm containers"
|
||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qE 'glm|frank'; then
|
||||
podman ps -a --format '{{.Names}}|{{.Status}}' 2>/dev/null \
|
||||
| awk -F'|' '$1 ~ /glm|frank/ {printf " %-18s %s\n",$1,$2}'
|
||||
else
|
||||
echo " none"
|
||||
fi
|
||||
|
||||
section "frank-primary unit"
|
||||
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
|
||||
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
|
||||
|
||||
section "glm serve tail"
|
||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
|
||||
podman logs "$GLM_CTR" 2>&1 | grep -viE 'multiproc_executor\.py:943' | tail -5 | sed 's/^/ /'
|
||||
else
|
||||
echo " (container $GLM_CTR not present)"
|
||||
fi
|
||||
|
||||
section "done"
|
||||
Reference in New Issue
Block a user