chore(llm): add dankstatus.sh — one verified parse path for Dankserver state

The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so

  grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2

returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.

dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.

Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.

Tests: 346/346 pass (no test surface touched).
This commit is contained in:
Frank committed 2026-09-28 23:27:50 -07:00
1 parent b4b6777e49
commit 32abc8d543
8 files changed
+525 -146

No files matched your search

+13 -9
View File
@@ -1096,7 +1096,7 @@ deploy_anubis() {
# same as every cutover since 2026-09-16: nothing starts until 7770 is
# released.
LLM_LARGE_RUNTIME="${LLM_LARGE_RUNTIME:-/home/dank/GitPackages/gufo-v0.1.1/bin/gufo}"
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf}"
LLM_LARGE_MODEL="${LLM_LARGE_MODEL:-/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
LLM_LARGE_DRAFT="${LLM_LARGE_DRAFT:-/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
retire_halogen_large_quadlet() {
@@ -1151,15 +1151,19 @@ deploy_llm_large() {
retire_halogen_large_quadlet
# On a re-deploy the predecessor is THIS unit: stop it before waiting on
# the port, or the wait deadlocks on the server it just stopped serving.
# Then sweep a stray manual run of the SAME model -- that is how a bench
# harness leaves the port busy, and the pattern matches the gufo command
# line because it carries the model path. Match the model path, not a
# binary name: frank-small on this host serves a different GGUF over
# llama-server and must not be killed. The pattern never appears in the
# deploying shell's own command line, so -f is safe here (it is not in
# `bash deploy-remote.sh`).
# Then sweep a stray manual run -- that is how a bench harness leaves the
# port busy. Match the DRAFTER path, which every frank-large command
# line carries via --dflash-model and which is UNCHANGED by the 2026-09-28
# payload swap: the target moved to a different repository while the
# drafter stayed put, so keying on the target filename would have quietly
# stopped matching. It is also the right key in principle — it names this
# endpoint's speculative drafter, not a binary. Match that, not a binary
# name: frank-small on this host serves a different GGUF over llama-server
# with no DFlash2 drafter and must not be killed. The pattern never
# appears in the deploying shell's own command line, so -f is safe here
# (it is not in `bash deploy-remote.sh`).
run_privileged systemctl stop frank-large.service 2>/dev/null || true
pkill -f 'Qwen3.8-27B-UD-Q8_K_XL' 2>/dev/null || true
pkill -f 'Qwen3.8-27B-DFlash2-Q4_K_M' 2>/dev/null || true
# Nothing may start until the predecessor has released the port. On
# 2026-09-16 the cutover failed exactly here: the old server logged
# "Deactivated successfully", the next engine tried to bind 7770 a second