73 lines
4.5 KiB
Markdown
73 lines
4.5 KiB
Markdown
---
|
|
name: qwen38-chain-rerun-dankserver
|
|
description: "Rerun the qwen38-w5nl64 campaign on dankserver: re-chunk the KLD corpus to the real KV pool, verify the rig's gate with --dry-run, deploy, and relaunch run-chained.sh detached — plus the three state-reading traps (root vs rootless podman, unit failed during an outage, the .rc sentinel)."
|
|
---
|
|
|
|
# Qwen38 campaign rerun on dankserver
|
|
|
|
Use when rerunning the `qwen38-w5nl64` campaign chain (kldref -> calib -> retire the reference `.rad`) on Dankserver, or when the chain died with `E kld: document N is <n> tokens and the KV pool holds <pool>` and needs its corpus fixed.
|
|
|
|
Order is fixed: **corpus -> gate -> deploy -> chain.** Skipping the gate burns the outage window.
|
|
|
|
## 1. Corpus must fit the pool, not the flag
|
|
|
|
The rig's `--max-model-len` is decorative: the recorder refuses any document longer than the **KV pool the boot actually got** (printed by the boot ledger, e.g. `7,436`), so a corpus authored at a wider window scores nothing.
|
|
|
|
```bash
|
|
ssh dankserver '/home/dank/venvs/radcorpus/bin/python \
|
|
/home/dank/campaign/qwen38-w5nl64/rechunk-kld-corpus.py \
|
|
--corpus /home/dank/corpora/out/kld.jsonl \
|
|
--out /home/dank/corpora/out/kld-7168.jsonl'
|
|
```
|
|
|
|
Read the summary, don't trust the run: `tokens_in` must equal `tokens_out` (that equality is the whole invariant — the tool exists to prove it), `longest_document` must equal the ceiling, and `scored_positions` grows because every window now carries its own `score_from`.
|
|
|
|
Chars-per-token is payload-specific: this file is chars/3.6, another corpus measured chars/4.97. Never size a window from chars alone.
|
|
|
|
## 2. Gate the rig before spending the outage
|
|
|
|
`scripts/run-qwen38-bf16.sh kldref --dry-run` runs the corpus gate and prints the argv **without touching the production unit**:
|
|
|
|
```bash
|
|
ssh dankserver 'cd /home/dank/serverAppConfigs && \
|
|
FRANK_QWEN38_KLD_CORPUS=/home/dank/corpora/out/kld-7168.jsonl \
|
|
bash scripts/run-qwen38-bf16.sh kldref --dry-run'
|
|
```
|
|
|
|
Check the printed line `corpus fits: longest record <chars> ... inside the <window>` **and** that the `-v ...:/data/kld.jsonl:ro` mount names the file you intended — the rig mounts whatever `FRANK_QWEN38_KLD_CORPUS` points at, and its default is the old file.
|
|
|
|
## 3. Deploy from a clean MAIN checkout, not a feature worktree
|
|
|
|
`deploy.sh` refuses off-main (`FAIL: deploy.sh must run from the main branch.`). Run it from the local main checkout and name the service — a bare target is a fleet-wide drain:
|
|
|
|
```bash
|
|
cd ~/Projects/serverAppConfig && ./scripts/deploy.sh dankserver --service frank-archon
|
|
```
|
|
|
|
If the remote pull aborts with `Your local changes ... would be overwritten by merge`, the remote copy is a committed blob written directly on the host. Prove it before discarding:
|
|
|
|
```bash
|
|
git -C /home/dank/serverAppConfigs hash-object <file> # working copy
|
|
git -C /home/dank/serverAppConfigs rev-parse <sha>:<file> # candidate commit
|
|
```
|
|
|
|
Equal -> `git checkout -- <file>` is lossless (the blob is in the object store), then re-deploy. Unequal -> archive to `.AGENTTRASH/<date>-<topic>/` first, then port into the repo.
|
|
|
|
## 4. Launch the chain detached, record the pid
|
|
|
|
```bash
|
|
ssh dankserver 'cd /home/dank/campaign/qwen38-w5nl64 && \
|
|
nohup bash run-chained.sh >> run-chained.log 2>&1 < /dev/null & \
|
|
echo "chain_pid=$!" > run-chained.pid'
|
|
```
|
|
|
|
- `run-chained.sh` is a server-side campaign artifact (not tracked in the repo); it chains the phases and must export `FRANK_QWEN38_KLD_CORPUS` for the re-chunked file.
|
|
- Move a prior `kld-ref/` into `.AGENTTRASH/<date>-qwen38-kldref-attemptN/` first, or the new recording's artefacts mix with the old ones.
|
|
- The chain writes `/home/dank/logs/qwen38-bf16-run.rc` as its completion sentinel; watch the pid, don't poll the log.
|
|
|
|
## 5. Reading state during the outage (three traps)
|
|
|
|
- **Container storage is split**: `frank-archon` is a root unit -> its container appears only under `sudo podman ps`; the media stack (romm, copyparty, exporters) is rootless -> only plain `podman ps`. Each view legitimately shows "nothing" for the other half. `pgrep -af radiance` settles which runtime is live.
|
|
- **`frank-archon` reads `failed` while the rig holds the card.** The rig stops the unit; the container exits status=1 on stop, so systemd marks `failed` until the phase restores it. Confirm with `journalctl -u frank-archon -n 6` — `Stopping...` then `Failed with result 'exit-code'` means the outage, not a crash. `systemctl show -p ActiveEnterTimestamp` dates it.
|
|
- **Engine stdout lives in `/home/dank/logs/`, not the journal** (for halogen); for radiance read the chain log.
|