diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index 40b4f57..8810a32 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -425,7 +425,32 @@ bash scripts/download-frank-primary.sh --verify-only # live telemetry (needs the profiler window; with_stack=false is mandatory — # with stack the stop path hangs and the trace buffer grows to tens of GB): podman logs frank-primary-glm 2>&1 | grep "expert_lru.py" +``` +**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine +**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state +`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a +liveness probe a lying oracle for a restore — twice a restore script printed +`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was +still in place. The restore oracle must be a **conjunction**: `is-active` == +`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/` +mask symlink **and** the container is owned by the unit and not `Stopping`. The +repair recipe, all four steps needed: + +```bash +sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start +sudo systemctl daemon-reload +sudo systemctl reset-failed frank-primary.service # a failed unit refuses to start +podman rm -f frank-primary-glm # SIGKILL; `podman stop` hangs on the same SIGTERM +sudo systemctl start frank-primary.service # then ~7 min to health 200 +``` + +Also: **never let two experiment jobs overlap.** A later job's `podman rm -f +glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its +container, so one whole round produced no evidence at all (`no container with name +glm-probe found`) — check `proc://` for running jobs before opening a window. + +```bash # boot: ~7 min warm page cache, longer cold; health returns 503 while loading curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health