From cfabe07959909eb1ccbad3d4a681d287439756b3 Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Fri, 18 Sep 2026 12:29:36 -0700
Subject: [PATCH] log: frank-medium restored after orphaned merge-build server
 held both R9700s

---
 serverAppConfigs-docs/log.md | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md
index 9bd7bbc..5c55db8 100644
--- a/serverAppConfigs-docs/log.md
+++ b/serverAppConfigs-docs/log.md
@@ -905,3 +905,39 @@ timestamp: 2026-09-14
   If you bench from an agent session, the session owes the fleet a
   `health 200` before it ends. Consider making the bench harness
   self-restoring.
+
+## 2026-09-18 (midday) — frank-medium down: orphaned merge-build server held both R9700s
+
+- **Symptom:** connection refused on `192.168.0.235:7770`. Unit was
+  `inactive`, `enabled`, unmasked (`/etc/systemd/system/frank-medium.service`
+  symlinked to the repo unit, not `/dev/null`), `NRestarts=0` — a clean
+  stop, not a crash.
+- **Not the morning's case:** the GPUs were **not** idle. `rocm-smi` read
+  15.5 GiB on GPU0 and 17.1 GiB on GPU1. `ps` named the holder: PID
+  2519965, `llama-server` from `third_party/llamacpp/build-merge`
+  (gitignored scratch), orphaned at PPID 1, bound `127.0.0.1:17770`,
+  `-c 16384 --split-mode layer --tensor-split 1,1 -ngl 99 -t 16`.
+- **Timeline (sudo journal + `/tmp/layersplit.log`):** `start` 08:14:02 →
+  `stop` 09:19:44 → merge build spawned 09:19:47 → log reached
+  `model loaded` / `listening` at 09:20 → nothing after. The merge-build
+  load validation passed; the cleanup-and-restore half never ran.
+- **Why prod could not come back:** the 35:30 split needs ~30.8 GiB on
+  ROCm0 and ~28.9 on ROCm1. The stray owned roughly half of each card, so
+  `systemctl start` would have died on allocation, not on config.
+- **Proof it was abandoned, not parked:** `/proc/<pid>/stat` utime+stime
+  identical across a 10 s sample (frozen CPU), zero established
+  connections to `:17770`, loopback-only bind, 3h05m since last log write.
+- **Restore:** preserved the test log to `/tmp/layersplit.log.preserved-20260918`,
+  `kill 2519965` (exited in <1 s, both cards → 60 MiB baseline), then
+  `systemctl start frank-medium`. Health 503 → 200 in ~20 s (payload warm
+  in page cache). `/proc/MainPID/cmdline` confirms deployed geometry
+  (`--tensor-split 35,30`, ctx 393216, 2 slots, `--flash-attn on`,
+  `--spec-type draft-mtp`); steady state ROCm0 32.9 GiB / ROCm1 30.7 GiB.
+  Real completion returns verbatim with `reasoning_content` and
+  spec-decode engaged: `draft_n 39 / accepted 35` (89.7 %), tg 40.6 t/s.
+  Fingerprint `b11009-fb27a525d` = the pinned runtime.
+- **Lesson:** "GPUs idle" is a check, not an assumption. A stopped prod
+  unit plus busy cards means something else took them — find the holder with
+  `ps`/`ss` and prove it abandoned (frozen CPU ticks, no conns) before
+  reclaiming. Scratch servers on loopback are invisible to the endpoint
+  monitor, so they strand the fleet instead of failing loudly.