From 62779aa23721295946277387b2aa684959d54519 Mon Sep 17 00:00:00 2001
From: Comrade Chief Designer <chief@dank.local>
Date: Wed, 30 Sep 2026 14:54:07 -0700
Subject: [PATCH] frank-primary: document the experiment-window trap that faked
 two successful restores
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This engine ignores SIGTERM: after systemctl stop its container sits in podman ps
state 'Stopping' for 20+ minutes WHILE STILL ANSWERING /health with 200. So a liveness
probe is a lying oracle for a window restore — twice a restore reported PROD HEALTHY
while systemctl is-active said failed and the runtime mask was still in place, because
the old container was still serving.

Records the conjunction the restore must verify (is-active == active AND is-enabled ==
enabled AND no /run/systemd/system/<unit> mask symlink AND the container is owned by
the unit and not Stopping) and the four-step repair recipe (mv the runtime mask away —
it survives stop and blocks start; daemon-reload; reset-failed — a failed unit refuses
to start; podman rm -f — podman stop hangs on the same SIGTERM).

Also records the overlap hazard: a later job's 'podman rm -f glm-probe' cleanup
deleted an in-flight job's container, so one compile round produced no evidence at all
('no container with name glm-probe found'). Check proc:// for running jobs before
opening a window.

tests/run.sh: 362 seams pass.
---
 .../LLM/frank-primary-glm53-flash-w4a16.md    | 25 +++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
index 40b4f57..8810a32 100644
--- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
+++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
@@ -425,7 +425,32 @@ bash scripts/download-frank-primary.sh --verify-only
 # live telemetry (needs the profiler window; with_stack=false is mandatory —
 # with stack the stop path hangs and the trace buffer grows to tens of GB):
 podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
+```
 
+**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
+**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
+`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
+liveness probe a lying oracle for a restore — twice a restore script printed
+`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
+still in place. The restore oracle must be a **conjunction**: `is-active` ==
+`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
+mask symlink **and** the container is owned by the unit and not `Stopping`. The
+repair recipe, all four steps needed:
+
+```bash
+sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
+sudo systemctl daemon-reload
+sudo systemctl reset-failed frank-primary.service                            # a failed unit refuses to start
+podman rm -f frank-primary-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
+sudo systemctl start frank-primary.service                                   # then ~7 min to health 200
+```
+
+Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
+glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
+container, so one whole round produced no evidence at all (`no container with name
+glm-probe found`) — check `proc://` for running jobs before opening a window.
+
+```bash
 # boot: ~7 min warm page cache, longer cold; health returns 503 while loading
 curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health