live: heartbeat sync 2026-10-28T11:37:39-07:00 (Refs #11)

This commit is contained in:
Frank committed 2026-10-28 11:37:39 -07:00
1 parent 38236f35f7
commit a071d71e99
4 files changed
+138 -2

No files matched your search

@@ -0,0 +1,64 @@
---
name: deploy-target-dirty-blob-audit
description: "When scripts/deploy.sh on serverAppConfigs aborts with \"Your local changes to the following files would be overwritten by merge\", prove whether the target's dirty file is already-merged content before restoring it, then deploy from the main checkout."
---
# Deploy target dirty-file audit
`deploy.sh <host>` pushes the local branch, then ssh-pulls on the target. The pull is
`git pull --ff-only origin main` on the TARGET checkout, so any uncommitted edit there
aborts the whole deploy with:
```
error: Your local changes to the following files would be overwritten by merge:
scripts/run-qwen38-bf16.sh
Aborting
```
Nothing is dispatched, so the service is untouched (no wasted outage) — but the deploy
cannot proceed until that path is reconciled. This happens when an agent edited the file
directly on the host (the repo's "never edit on the remote" law) and committed the same
bytes later from a worktree, leaving the host's HEAD behind its own working tree.
## 1. Classify before touching anything
```bash
ssh <host> 'cd <checkout> && git status --porcelain' # every dirty path
ssh <host> 'cd <checkout> && git hash-object <file>
&& git rev-parse <target_sha>:<file>' # compare blobs
```
- Hash equal to the blob the pull wants (`origin/main:<file>`, or a commit main contains)
→ the edit is already merged; restoring is lossless, no trash copy needed.
- Hash matches neither HEAD nor the incoming blob → real unmerged content: `cp` it to
`.AGENTTRASH/YYYYMMDD-topic/` first, then decide.
- `D` entries under `.AGENTTRASH/` (deletions, not modifications) do not block the pull
unless those paths changed between HEAD and main. Leave them — only Frank purges trash.
## 2. Restore, then deploy from the MAIN checkout
```bash
ssh <host> 'cd <checkout> && git checkout -- <file>'
cd ~/Projects/serverAppConfig && ./scripts/deploy.sh <host> --service <svc>
```
`deploy.sh` refuses with `FAIL: deploy.sh must run from the main branch.` when run from a
feature worktree — use the host's `~/Projects/serverAppConfig` main checkout, and pull it
`--ff-only` first so `[1/2] Pushing main to origin` reports `Everything up-to-date`.
A missing server-side path (`/home/dank/models`) plus an `rm -f` in the chain means the
`.rad` was already retired; confirm before relaunching anything that needs it.
## 3. Long deploys never fit an async bash slot
An LLM-service deploy waits up to 20 min for model load plus the Playwright gate, but an
async `bash` call is killed at 600 s. Launch it as a named service instead, then watch the
pid with a deadline-free watcher:
```bash
bash command="cd ~/Projects/serverAppConfig && ./scripts/deploy.sh <host> --service <svc>" \
name=deploy-<host> ready.log="Deployment plan"
# then: while kill -0 <pid> 2>/dev/null; do sleep 20; done (timeout 0, async)
```
`ready.log="Deployment plan"` matches the plan banner, so readiness reports the deploy
started rather than finished; the real result arrives when the watcher completes.
@@ -0,0 +1,72 @@
---
name: qwen38-chain-rerun-dankserver
description: "Rerun the qwen38-w5nl64 campaign on dankserver: re-chunk the KLD corpus to the real KV pool, verify the rig's gate with --dry-run, deploy, and relaunch run-chained.sh detached — plus the three state-reading traps (root vs rootless podman, unit failed during an outage, the .rc sentinel)."
---
# Qwen38 campaign rerun on dankserver
Use when rerunning the `qwen38-w5nl64` campaign chain (kldref -> calib -> retire the reference `.rad`) on Dankserver, or when the chain died with `E kld: document N is <n> tokens and the KV pool holds <pool>` and needs its corpus fixed.
Order is fixed: **corpus -> gate -> deploy -> chain.** Skipping the gate burns the outage window.
## 1. Corpus must fit the pool, not the flag
The rig's `--max-model-len` is decorative: the recorder refuses any document longer than the **KV pool the boot actually got** (printed by the boot ledger, e.g. `7,436`), so a corpus authored at a wider window scores nothing.
```bash
ssh dankserver '/home/dank/venvs/radcorpus/bin/python \
/home/dank/campaign/qwen38-w5nl64/rechunk-kld-corpus.py \
--corpus /home/dank/corpora/out/kld.jsonl \
--out /home/dank/corpora/out/kld-7168.jsonl'
```
Read the summary, don't trust the run: `tokens_in` must equal `tokens_out` (that equality is the whole invariant — the tool exists to prove it), `longest_document` must equal the ceiling, and `scored_positions` grows because every window now carries its own `score_from`.
Chars-per-token is payload-specific: this file is chars/3.6, another corpus measured chars/4.97. Never size a window from chars alone.
## 2. Gate the rig before spending the outage
`scripts/run-qwen38-bf16.sh kldref --dry-run` runs the corpus gate and prints the argv **without touching the production unit**:
```bash
ssh dankserver 'cd /home/dank/serverAppConfigs && \
FRANK_QWEN38_KLD_CORPUS=/home/dank/corpora/out/kld-7168.jsonl \
bash scripts/run-qwen38-bf16.sh kldref --dry-run'
```
Check the printed line `corpus fits: longest record <chars> ... inside the <window>` **and** that the `-v ...:/data/kld.jsonl:ro` mount names the file you intended — the rig mounts whatever `FRANK_QWEN38_KLD_CORPUS` points at, and its default is the old file.
## 3. Deploy from a clean MAIN checkout, not a feature worktree
`deploy.sh` refuses off-main (`FAIL: deploy.sh must run from the main branch.`). Run it from the local main checkout and name the service — a bare target is a fleet-wide drain:
```bash
cd ~/Projects/serverAppConfig && ./scripts/deploy.sh dankserver --service frank-archon
```
If the remote pull aborts with `Your local changes ... would be overwritten by merge`, the remote copy is a committed blob written directly on the host. Prove it before discarding:
```bash
git -C /home/dank/serverAppConfigs hash-object <file> # working copy
git -C /home/dank/serverAppConfigs rev-parse <sha>:<file> # candidate commit
```
Equal -> `git checkout -- <file>` is lossless (the blob is in the object store), then re-deploy. Unequal -> archive to `.AGENTTRASH/<date>-<topic>/` first, then port into the repo.
## 4. Launch the chain detached, record the pid
```bash
ssh dankserver 'cd /home/dank/campaign/qwen38-w5nl64 && \
nohup bash run-chained.sh >> run-chained.log 2>&1 < /dev/null & \
echo "chain_pid=$!" > run-chained.pid'
```
- `run-chained.sh` is a server-side campaign artifact (not tracked in the repo); it chains the phases and must export `FRANK_QWEN38_KLD_CORPUS` for the re-chunked file.
- Move a prior `kld-ref/` into `.AGENTTRASH/<date>-qwen38-kldref-attemptN/` first, or the new recording's artefacts mix with the old ones.
- The chain writes `/home/dank/logs/qwen38-bf16-run.rc` as its completion sentinel; watch the pid, don't poll the log.
## 5. Reading state during the outage (three traps)
- **Container storage is split**: `frank-archon` is a root unit -> its container appears only under `sudo podman ps`; the media stack (romm, copyparty, exporters) is rootless -> only plain `podman ps`. Each view legitimately shows "nothing" for the other half. `pgrep -af radiance` settles which runtime is live.
- **`frank-archon` reads `failed` while the rig holds the card.** The rig stops the unit; the container exits status=1 on stop, so systemd marks `failed` until the phase restores it. Confirm with `journalctl -u frank-archon -n 6` — `Stopping...` then `Failed with result 'exit-code'` means the outage, not a crash. `systemctl show -p ActiveEnterTimestamp` dates it.
- **Engine stdout lives in `/home/dank/logs/`, not the journal** (for halogen); for radiance read the chain log.
@@ -1,6 +1,6 @@
{ {
"v": 1, "v": 1,
"lastConsolidatedAt": 1793208636702, "lastConsolidatedAt": 1793212656722,
"lastResult": { "lastResult": {
"at": 1793170367714, "at": 1793170367714,
"sessions": 1, "sessions": 1,
@@ -285,7 +285,7 @@ transition = [ "fade" ]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
[wallpaper.monitors.DP-1] [wallpaper.monitors.DP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
[wallpaper.monitors.DP-2] [wallpaper.monitors.DP-2]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"