diff --git a/AGENTS.md b/AGENTS.md index 99f3e44..461a485 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,7 +85,7 @@ graph TD - `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '' | sudo -S `. - Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md). - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. -- Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. +- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`). - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. diff --git a/scripts/deploy.sh b/scripts/deploy.sh index 4d4b564..2dc6019 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -31,7 +31,7 @@ set -euo pipefail # --------------------------------------------------------------------------- # Target configuration # --------------------------------------------------------------------------- -declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH DROP_CACHES HELPER +declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH DROP_CACHES HELPER HEALTH_WAIT REMOTE_USER[dankclaw]=dank REMOTE_HOST[dankclaw]=dankclaw @@ -44,6 +44,9 @@ DROP_CACHES[dankclaw]=1 # Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-large). # When present, deploy.sh uses it instead of inline sudo commands. HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh" +# Health wait in iterations of 2s. Cold loads after the cache drop measured +# at up to 17 minutes on 2026-08-13; 600 iterations = 20 minutes. +HEALTH_WAIT[dankclaw]=600 REMOTE_USER[dankserver]=dank REMOTE_HOST[dankserver]=dankserver @@ -52,6 +55,7 @@ SERVICE[dankserver]="frank-default-small" SERVICE_HEALTH[dankserver]="http://127.0.0.1:7770/health" DROP_CACHES[dankserver]=0 HELPER[dankserver]="/usr/local/sbin/frank-restart-small.sh" +HEALTH_WAIT[dankserver]=300 TARGET="${1:-}" DRY_RUN=0 @@ -177,7 +181,8 @@ if [ "$DRY_RUN" = "1" ]; then else echo "[4/4] Waiting for health endpoint ${SERVICE_HEALTH[$TARGET]} (via ssh)..." HEALTH_URL="${SERVICE_HEALTH[$TARGET]}" - for i in $(seq 1 300); do + HEALTH_ITERS="${HEALTH_WAIT[$TARGET]:-300}" + for i in $(seq 1 "$HEALTH_ITERS"); do if ssh "$REMOTE" "curl -sf '$HEALTH_URL'" > /dev/null 2>&1; then echo " Service healthy after approx $((i * 2))s" echo "" @@ -186,6 +191,6 @@ else fi sleep 2 done - echo "FAIL: Service did not become healthy in 600s" + echo "FAIL: Service did not become healthy in $((HEALTH_ITERS * 2))s" exit 1 fi