dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: document the experiment-window trap that faked two successful restores

Browse Source 
This engine ignores SIGTERM: after systemctl stop its container sits in podman ps
state 'Stopping' for 20+ minutes WHILE STILL ANSWERING /health with 200. So a liveness
probe is a lying oracle for a window restore — twice a restore reported PROD HEALTHY
while systemctl is-active said failed and the runtime mask was still in place, because
the old container was still serving.

Records the conjunction the restore must verify (is-active == active AND is-enabled ==
enabled AND no /run/systemd/system/<unit> mask symlink AND the container is owned by
the unit and not Stopping) and the four-step repair recipe (mv the runtime mask away —
it survives stop and blocks start; daemon-reload; reset-failed — a failed unit refuses
to start; podman rm -f — podman stop hangs on the same SIGTERM).

Also records the overlap hazard: a later job's 'podman rm -f glm-probe' cleanup
deleted an in-flight job's container, so one compile round produced no evidence at all
('no container with name glm-probe found'). Check proc:// for running jobs before
opening a window.

tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 14:54:07 -07:00
parent 71db0debcc
commit 62779aa237
1 changed files with 25 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md
+25
View File
@@ -425,7 +425,32 @@ bash scripts/download-frank-primary.sh --verify-only
# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
```
**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
liveness probe a lying oracle for a restore — twice a restore script printed
`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
still in place. The restore oracle must be a **conjunction**: `is-active` ==
`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
mask symlink **and** the container is owned by the unit and not `Stopping`. The
repair recipe, all four steps needed:
```bash
sudo mv /run/systemd/system/frank-primary.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
sudo systemctl daemon-reload
sudo systemctl reset-failed frank-primary.service # a failed unit refuses to start
podman rm -f frank-primary-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
sudo systemctl start frank-primary.service # then ~7 min to health 200
```
Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
container, so one whole round produced no evidence at all (`no container with name
glm-probe found`) — check `proc://` for running jobs before opening a window.
```bash
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health