dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

deploy.sh: per-target health wait; 20 min for Dankclaw cold loads

Browse Source 
Cold loads after the cache drop measured 17 minutes on 2026-08-13, longer
than the previous 600s wait. HEALTH_WAIT[dankclaw]=600 iterations (20 min),
dankserver stays at 300 (10 min). AGENTS.md: update cold-load figure.
This commit is contained in:
Frank
2026-08-13 21:01:01 -07:00
parent e29071caa1
commit 42558f1707
2 changed files with 9 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1 -1
View File
@@ -85,7 +85,7 @@ graph TD
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process.
- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
scripts/deploy.sh
+8 -3
View File
@@ -31,7 +31,7 @@ set -euo pipefail
# ---------------------------------------------------------------------------
# Target configuration
# ---------------------------------------------------------------------------
declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH DROP_CACHES HELPER
declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH DROP_CACHES HELPER HEALTH_WAIT
REMOTE_USER[dankclaw]=dank
REMOTE_HOST[dankclaw]=dankclaw
@@ -44,6 +44,9 @@ DROP_CACHES[dankclaw]=1
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-large).
# When present, deploy.sh uses it instead of inline sudo commands.
HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh"
# Health wait in iterations of 2s. Cold loads after the cache drop measured
# at up to 17 minutes on 2026-08-13; 600 iterations = 20 minutes.
HEALTH_WAIT[dankclaw]=600
REMOTE_USER[dankserver]=dank
REMOTE_HOST[dankserver]=dankserver
@@ -52,6 +55,7 @@ SERVICE[dankserver]="frank-default-small"
SERVICE_HEALTH[dankserver]="http://127.0.0.1:7770/health"
DROP_CACHES[dankserver]=0
HELPER[dankserver]="/usr/local/sbin/frank-restart-small.sh"
HEALTH_WAIT[dankserver]=300
TARGET="${1:-}"
DRY_RUN=0
@@ -177,7 +181,8 @@ if [ "$DRY_RUN" = "1" ]; then
else
echo "[4/4] Waiting for health endpoint ${SERVICE_HEALTH[$TARGET]} (via ssh)..."
HEALTH_URL="${SERVICE_HEALTH[$TARGET]}"
for i in $(seq 1 300); do
HEALTH_ITERS="${HEALTH_WAIT[$TARGET]:-300}"
for i in $(seq 1 "$HEALTH_ITERS"); do
if ssh "$REMOTE" "curl -sf '$HEALTH_URL'" > /dev/null 2>&1; then
echo " Service healthy after approx $((i * 2))s"
echo ""
@@ -186,6 +191,6 @@ else
fi
sleep 2
done
echo "FAIL: Service did not become healthy in 600s"
echo "FAIL: Service did not become healthy in $((HEALTH_ITERS * 2))s"
exit 1
fi