live: heartbeat sync 2026-10-28T11:37:39-07:00 (Refs #11)

2026-10-28 11:37:39 -07:00
1 parent 38236f35f7
commit a071d71e99
@@ -0,0 +1,64 @@
+---
+name: deploy-target-dirty-blob-audit
+description: "When scripts/deploy.sh on serverAppConfigs aborts with \"Your local changes to the following files would be overwritten by merge\", prove whether the target's dirty file is already-merged content before restoring it, then deploy from the main checkout."
+---
+
+# Deploy target dirty-file audit
+
+`deploy.sh <host>` pushes the local branch, then ssh-pulls on the target. The pull is
+`git pull --ff-only origin main` on the TARGET checkout, so any uncommitted edit there
+aborts the whole deploy with:
+
+```
+error: Your local changes to the following files would be overwritten by merge:
+        scripts/run-qwen38-bf16.sh
+Aborting
+```
+
+Nothing is dispatched, so the service is untouched (no wasted outage) — but the deploy
+cannot proceed until that path is reconciled. This happens when an agent edited the file
+directly on the host (the repo's "never edit on the remote" law) and committed the same
+bytes later from a worktree, leaving the host's HEAD behind its own working tree.
+
+## 1. Classify before touching anything
+
+```bash
+ssh <host> 'cd <checkout> && git status --porcelain'          # every dirty path
+ssh <host> 'cd <checkout> && git hash-object <file>
+                       && git rev-parse <target_sha>:<file>'  # compare blobs
+```
+
+- Hash equal to the blob the pull wants (`origin/main:<file>`, or a commit main contains)
+  → the edit is already merged; restoring is lossless, no trash copy needed.
+- Hash matches neither HEAD nor the incoming blob → real unmerged content: `cp` it to
+  `.AGENTTRASH/YYYYMMDD-topic/` first, then decide.
+- `D` entries under `.AGENTTRASH/` (deletions, not modifications) do not block the pull
+  unless those paths changed between HEAD and main. Leave them — only Frank purges trash.
+
+## 2. Restore, then deploy from the MAIN checkout
+
+```bash
+ssh <host> 'cd <checkout> && git checkout -- <file>'
+cd ~/Projects/serverAppConfig && ./scripts/deploy.sh <host> --service <svc>
+```
+
+`deploy.sh` refuses with `FAIL: deploy.sh must run from the main branch.` when run from a
+feature worktree — use the host's `~/Projects/serverAppConfig` main checkout, and pull it
+`--ff-only` first so `[1/2] Pushing main to origin` reports `Everything up-to-date`.
+A missing server-side path (`/home/dank/models`) plus an `rm -f` in the chain means the
+`.rad` was already retired; confirm before relaunching anything that needs it.
+
+## 3. Long deploys never fit an async bash slot
+
+An LLM-service deploy waits up to 20 min for model load plus the Playwright gate, but an
+async `bash` call is killed at 600 s. Launch it as a named service instead, then watch the
+pid with a deadline-free watcher:
+
+```bash
+bash command="cd ~/Projects/serverAppConfig && ./scripts/deploy.sh <host> --service <svc>" \
+     name=deploy-<host> ready.log="Deployment plan"
+# then: while kill -0 <pid> 2>/dev/null; do sleep 20; done   (timeout 0, async)
+```
+
+`ready.log="Deployment plan"` matches the plan banner, so readiness reports the deploy
+started rather than finished; the real result arrives when the watcher completes.
@@ -0,0 +1,72 @@
+---
+name: qwen38-chain-rerun-dankserver
+description: "Rerun the qwen38-w5nl64 campaign on dankserver: re-chunk the KLD corpus to the real KV pool, verify the rig's gate with --dry-run, deploy, and relaunch run-chained.sh detached — plus the three state-reading traps (root vs rootless podman, unit failed during an outage, the .rc sentinel)."
+---
+
+# Qwen38 campaign rerun on dankserver
+
+Use when rerunning the `qwen38-w5nl64` campaign chain (kldref -> calib -> retire the reference `.rad`) on Dankserver, or when the chain died with `E kld: document N is <n> tokens and the KV pool holds <pool>` and needs its corpus fixed.
+
+Order is fixed: **corpus -> gate -> deploy -> chain.** Skipping the gate burns the outage window.
+
+## 1. Corpus must fit the pool, not the flag
+
+The rig's `--max-model-len` is decorative: the recorder refuses any document longer than the **KV pool the boot actually got** (printed by the boot ledger, e.g. `7,436`), so a corpus authored at a wider window scores nothing.
+
+```bash
+ssh dankserver '/home/dank/venvs/radcorpus/bin/python \
+  /home/dank/campaign/qwen38-w5nl64/rechunk-kld-corpus.py \
+  --corpus /home/dank/corpora/out/kld.jsonl \
+  --out    /home/dank/corpora/out/kld-7168.jsonl'
+```
+
+Read the summary, don't trust the run: `tokens_in` must equal `tokens_out` (that equality is the whole invariant — the tool exists to prove it), `longest_document` must equal the ceiling, and `scored_positions` grows because every window now carries its own `score_from`.
+
+Chars-per-token is payload-specific: this file is chars/3.6, another corpus measured chars/4.97. Never size a window from chars alone.
+
+## 2. Gate the rig before spending the outage
+
+`scripts/run-qwen38-bf16.sh kldref --dry-run` runs the corpus gate and prints the argv **without touching the production unit**:
+
+```bash
+ssh dankserver 'cd /home/dank/serverAppConfigs && \
+  FRANK_QWEN38_KLD_CORPUS=/home/dank/corpora/out/kld-7168.jsonl \
+  bash scripts/run-qwen38-bf16.sh kldref --dry-run'
+```
+
+Check the printed line `corpus fits: longest record <chars> ... inside the <window>` **and** that the `-v ...:/data/kld.jsonl:ro` mount names the file you intended — the rig mounts whatever `FRANK_QWEN38_KLD_CORPUS` points at, and its default is the old file.
+
+## 3. Deploy from a clean MAIN checkout, not a feature worktree
+
+`deploy.sh` refuses off-main (`FAIL: deploy.sh must run from the main branch.`). Run it from the local main checkout and name the service — a bare target is a fleet-wide drain:
+
+```bash
+cd ~/Projects/serverAppConfig && ./scripts/deploy.sh dankserver --service frank-archon
+```
+
+If the remote pull aborts with `Your local changes ... would be overwritten by merge`, the remote copy is a committed blob written directly on the host. Prove it before discarding:
+
+```bash
+git -C /home/dank/serverAppConfigs hash-object <file>            # working copy
+git -C /home/dank/serverAppConfigs rev-parse <sha>:<file>          # candidate commit
+```
+
+Equal -> `git checkout -- <file>` is lossless (the blob is in the object store), then re-deploy. Unequal -> archive to `.AGENTTRASH/<date>-<topic>/` first, then port into the repo.
+
+## 4. Launch the chain detached, record the pid
+
+```bash
+ssh dankserver 'cd /home/dank/campaign/qwen38-w5nl64 && \
+  nohup bash run-chained.sh >> run-chained.log 2>&1 < /dev/null & \
+  echo "chain_pid=$!" > run-chained.pid'
+```
+
+- `run-chained.sh` is a server-side campaign artifact (not tracked in the repo); it chains the phases and must export `FRANK_QWEN38_KLD_CORPUS` for the re-chunked file.
+- Move a prior `kld-ref/` into `.AGENTTRASH/<date>-qwen38-kldref-attemptN/` first, or the new recording's artefacts mix with the old ones.
+- The chain writes `/home/dank/logs/qwen38-bf16-run.rc` as its completion sentinel; watch the pid, don't poll the log.
+
+## 5. Reading state during the outage (three traps)
+
+- **Container storage is split**: `frank-archon` is a root unit -> its container appears only under `sudo podman ps`; the media stack (romm, copyparty, exporters) is rootless -> only plain `podman ps`. Each view legitimately shows "nothing" for the other half. `pgrep -af radiance` settles which runtime is live.
+- **`frank-archon` reads `failed` while the rig holds the card.** The rig stops the unit; the container exits status=1 on stop, so systemd marks `failed` until the phase restores it. Confirm with `journalctl -u frank-archon -n 6` — `Stopping...` then `Failed with result 'exit-code'` means the outage, not a crash. `systemctl show -p ActiveEnterTimestamp` dates it.
+- **Engine stdout lives in `/home/dank/logs/`, not the journal** (for halogen); for radiance read the chain log.
@@ -1,6 +1,6 @@
 {
 	"v": 1,
-	"lastConsolidatedAt": 1793208636702,
+	"lastConsolidatedAt": 1793212656722,
 	"lastResult": {
 		"at": 1793170367714,
 		"sessions": 1,