Frank
0aafad5f2c
The KLD engine exited clean with the record complete (kld.json rows: 50235, 24 GiB, 5 files), but the wrapper died rc=127: a mid-run scp of the calib-shape edit shifted bash's lazy-read offset — the fleet's own script-swap law, biting on the same class as the autoresearch collision. Resume path: SKIP_KLD=1 verifies the receipt (rows + size + files) and goes straight to calib. Never scp a script a running process owns.
40 lines
1.8 KiB
Bash
Executable File
40 lines
1.8 KiB
Bash
Executable File
#!/bin/bash
|
|
# kld-then-calib.sh — RUNS ON DANKSERVER. Campaign chain: wait for the KLD
|
|
# record to finish, verify it, then boot the 1.2.0 calibration server and
|
|
# push the 10M-token corpus. One GPU, so the phases serialize; prod is
|
|
# restored by each run script's trap.
|
|
set -u
|
|
KLOG=/home/dank/logs/swift15-kld-driver.log
|
|
CLOG=/home/dank/logs/swift15-calib-driver.log
|
|
# Phase 1: KLD record (SKIP_KLD=1 when the record is already complete —
|
|
# kld.json's rows count is the receipt; a mid-run script swap can kill the
|
|
# wrapper with rc=127 after the engine exited clean).
|
|
if [ "${SKIP_KLD:-0}" != 1 ]; then
|
|
while ! grep -q "exited rc=" "$KLOG" 2>/dev/null; do sleep 60; done
|
|
RC=$(grep -o "exited rc=[0-9]*" "$KLOG" | tail -1)
|
|
if [ "$RC" != "exited rc=0" ]; then echo "KLD FAILED ($RC); calib NOT started"; exit 1; fi
|
|
fi
|
|
N=$(find /home/dank/corpora/kld-ref -type f | wc -l)
|
|
SZ=$(du -sm /home/dank/corpora/kld-ref | cut -f1)
|
|
echo "KLD OK files=$N size_mib=$SZ"
|
|
[ "$N" -ge 5 ] && [ "$SZ" -ge 1000 ] || { echo "kld-ref looks thin; calib NOT started"; exit 1; }
|
|
grep -q '"rows": 50235' /home/dank/corpora/kld-ref/kld.json || {
|
|
echo "kld.json rows != 50235; calib NOT started"; exit 1; }
|
|
|
|
# Phase 2: calibration serve (1.2.0) + corpus push.
|
|
nohup bash "$S/run-swift15-bf16.sh" calib > "$CLOG" 2>&1 &
|
|
CP=$!
|
|
for _ in $(seq 1 160); do
|
|
sleep 15
|
|
curl -sf localhost:7770/health >/dev/null 2>&1 && break
|
|
done
|
|
if ! curl -sf localhost:7770/health >/dev/null 2>&1; then
|
|
echo "CALIB BOOT FAILED; see /home/dank/logs/swift15_bf16.log"; kill "$CP" 2>/dev/null; exit 1
|
|
fi
|
|
echo "CALIB UP, pushing corpus"
|
|
python3 "$S/calib-push.py" --port 7770 --parallel 16 --container swift15ref --max-rounds 6
|
|
PUSH=$?
|
|
kill "$CP" 2>/dev/null
|
|
sleep 10
|
|
echo "CALIB_CHAIN_DONE push=$PUSH grams=$(find /home/dank/corpora/calib -name 'gram.r*.bin' 2>/dev/null | wc -l)"
|