dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

deploy.sh: drop page caches before restart; headless sudo; ssh-based health check

Browse Source 
Deterministic deployment instead of agent ritual:

- DROP_CACHES per target (dankclaw=1): runs 'sync; echo 3 >
  /proc/sys/vm/drop_caches' before systemctl restart. Without it the
  model load stalls at 28 GiB VRAM when the model pages are cached.
- FRANK_SUDO_PW env var: remote sudo pipes it via 'sudo -S' so the
  script runs headless. Falls back to plain sudo otherwise.
- Health check runs curl on the target over ssh (127.0.0.1:7770 was
  previously checked from the workstation, which can never see it).
This commit is contained in:
Frank
2026-08-13 19:05:26 -07:00
parent 317610a540
commit c96cb6a529
3 changed files with 52 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1 -1
View File
@@ -85,7 +85,7 @@ graph TD
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, more than 8 minutes cold. Health returns `503 Loading model` the whole time. That is normal. Do not kill the process.
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing.
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
### Dankclaw memory model
docs/deployment.md
+10 -5
View File
@@ -59,18 +59,22 @@ The script does these actions:
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Installs or updates the systemd unit.
5. Reloads systemd and restarts the service.
6. Waits for the health endpoint to respond.
5. Drops page caches on Dankclaw before the restart. This prevents the load stall that happens when the model's pages are fully cached.
6. Reloads systemd and restarts the service.
7. Waits for the health endpoint to respond (over ssh).
## Manual Update
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the password from stdin:
If you cannot run the deploy script, update a server manually. Complete each step on the target machine.
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```
### Update Dankclaw
1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Reload and restart the service.
3. Drop the page caches.
4. Reload and restart the service.
Run these commands on Dankclaw:
@@ -78,6 +82,7 @@ Run these commands on Dankclaw:
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
scripts/deploy.sh
+41 -12
View File
@@ -5,14 +5,18 @@
# AGENTS.md. It assumes the local changes are committed and pushed, then:
# 1. Pulls the repo on the target
# 2. Installs / enables the systemd unit
# 3. Reloads systemd and restarts the service
# 3. Drops page caches (dankclaw) and restarts the service
# 4. Waits for the health endpoint and reports status
#
# Sudo: when FRANK_SUDO_PW is set, remote sudo pipes it via `sudo -S` for
# headless runs. Otherwise plain `sudo` is used (tty or passwordless config).
#
# Usage:
# ./deploy.sh <target> [--dry-run]
# FRANK_SUDO_PW=... ./deploy.sh <target>
#
# Targets:
# dankclaw - vLLM-ROCm server (frank-default-large.service, port 7770)
# dankclaw - llama.cpp server (frank-default-large.service, port 7770)
# dankserver - llama.cpp server (frank-default-small.service, port 7770)
#
# Example:
@@ -23,19 +27,23 @@ set -euo pipefail
# ---------------------------------------------------------------------------
# Target configuration
# ---------------------------------------------------------------------------
declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH
declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH DROP_CACHES
REMOTE_USER[dankclaw]=dank
REMOTE_HOST[dankclaw]=dankclaw
REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
SERVICE[dankclaw]="frank-default-large"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
# Drop page caches before restart: with the model's pages fully cached, the
# KV/graph allocation stalls at load (VRAM stuck at 28 GiB). See AGENTS.md.
DROP_CACHES[dankclaw]=1
REMOTE_USER[dankserver]=dank
REMOTE_HOST[dankserver]=dankserver
REMOTE_DIR[dankserver]="/home/dank/vLLM"
SERVICE[dankserver]="frank-default-small"
SERVICE_HEALTH[dankserver]="http://127.0.0.1:7770/health"
DROP_CACHES[dankserver]=0
TARGET="${1:-}"
DRY_RUN=0
@@ -55,6 +63,16 @@ if [ -z "${REMOTE_HOST[$TARGET]:-}" ]; then
exit 1
fi
# ---------------------------------------------------------------------------
# Sudo wrapper (headless-friendly)
# ---------------------------------------------------------------------------
# When FRANK_SUDO_PW is set, remote sudo reads the password from stdin via
# `sudo -S`. Otherwise plain `sudo` is used (tty or passwordless config).
SUDO_PREFIX="sudo"
if [ -n "${FRANK_SUDO_PW:-}" ]; then
SUDO_PREFIX="printf '%s\n' '$FRANK_SUDO_PW' | sudo -S"
fi
# ---------------------------------------------------------------------------
# Local checks (pre-flight)
# ---------------------------------------------------------------------------
@@ -93,6 +111,14 @@ REMOTE="$(
# ---------------------------------------------------------------------------
# Remote: pull + install + restart
# ---------------------------------------------------------------------------
DROP_CACHES_CMD=""
if [ "${DROP_CACHES[$TARGET]}" = "1" ]; then
DROP_CACHES_CMD="
echo 'Dropping page caches for reliable model load...'
${SUDO_PREFIX} sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
"
fi
REMOTE_SCRIPT="
set -euo pipefail
cd '${REMOTE_DIR[$TARGET]}'
@@ -103,24 +129,27 @@ git pull origin main
# Install / symlink the systemd unit
case '${TARGET}' in
dankclaw)
sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-large.service' /etc/systemd/system/frank-default-large.service
${SUDO_PREFIX} ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-large.service' /etc/systemd/system/frank-default-large.service
;;
dankserver)
sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-small.service' /etc/systemd/system/frank-default-small.service
${SUDO_PREFIX} ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-small.service' /etc/systemd/system/frank-default-small.service
;;
esac
${DROP_CACHES_CMD}
echo '[3/4] Reload and restart ${SERVICE[$TARGET]}...'
sudo systemctl daemon-reload
sudo systemctl enable '${SERVICE[$TARGET]}'
sudo systemctl restart '${SERVICE[$TARGET]}'
${SUDO_PREFIX} systemctl daemon-reload
${SUDO_PREFIX} systemctl enable '${SERVICE[$TARGET]}'
${SUDO_PREFIX} systemctl restart '${SERVICE[$TARGET]}'
"
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Remote commands would run:"
echo "$REMOTE_SCRIPT"
else
ssh "$REMOTE" "bash -c '$REMOTE_SCRIPT'"
# shellcheck disable=SC2087
ssh "$REMOTE" bash -s <<REMOTE_EOF
${REMOTE_SCRIPT}
REMOTE_EOF
fi
# ---------------------------------------------------------------------------
@@ -129,10 +158,10 @@ fi
if [ "$DRY_RUN" = "1" ]; then
echo "[dry-run] Would wait for health at ${SERVICE_HEALTH[$TARGET]}"
else
echo "[4/4] Waiting for health endpoint ${SERVICE_HEALTH[$TARGET]}..."
echo "[4/4] Waiting for health endpoint ${SERVICE_HEALTH[$TARGET]} (via ssh)..."
HEALTH_URL="${SERVICE_HEALTH[$TARGET]}"
for i in $(seq 1 300); do
if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then
if ssh "$REMOTE" "curl -sf '$HEALTH_URL'" > /dev/null 2>&1; then
echo " Service healthy after approx $((i * 2))s"
echo ""
echo "=== Deploy complete: ${TARGET} ==="