---
name: qwen38-chain-rerun-dankserver
description: "Rerun the qwen38-w5nl64 campaign on dankserver: re-chunk the KLD corpus to the real KV pool, verify the rig's gate with --dry-run, deploy, and relaunch run-chained.sh detached — plus the three state-reading traps (root vs rootless podman, unit failed during an outage, the .rc sentinel)."
---

# Qwen38 campaign rerun on dankserver

Use when rerunning the `qwen38-w5nl64` campaign chain (kldref -> calib -> retire the reference `.rad`) on Dankserver, or when the chain died with `E kld: document N is <n> tokens and the KV pool holds <pool>` and needs its corpus fixed.

Order is fixed: **corpus -> gate -> deploy -> chain.** Skipping the gate burns the outage window.

## 1. Corpus must fit the pool, not the flag

The rig's `--max-model-len` is decorative: the recorder refuses any document longer than the **KV pool the boot actually got** (printed by the boot ledger, e.g. `7,436`), so a corpus authored at a wider window scores nothing.

```bash
ssh dankserver '/home/dank/venvs/radcorpus/bin/python \
  /home/dank/campaign/qwen38-w5nl64/rechunk-kld-corpus.py \
  --corpus /home/dank/corpora/out/kld.jsonl \
  --out    /home/dank/corpora/out/kld-7168.jsonl'
```

Read the summary, don't trust the run: `tokens_in` must equal `tokens_out` (that equality is the whole invariant — the tool exists to prove it), `longest_document` must equal the ceiling, and `scored_positions` grows because every window now carries its own `score_from`.

Chars-per-token is payload-specific: this file is chars/3.6, another corpus measured chars/4.97. Never size a window from chars alone.

## 2. Gate the rig before spending the outage

`scripts/run-qwen38-bf16.sh kldref --dry-run` runs the corpus gate and prints the argv **without touching the production unit**:

```bash
ssh dankserver 'cd /home/dank/serverAppConfigs && \
  FRANK_QWEN38_KLD_CORPUS=/home/dank/corpora/out/kld-7168.jsonl \
  bash scripts/run-qwen38-bf16.sh kldref --dry-run'
```

Check the printed line `corpus fits: longest record <chars> ... inside the <window>` **and** that the `-v ...:/data/kld.jsonl:ro` mount names the file you intended — the rig mounts whatever `FRANK_QWEN38_KLD_CORPUS` points at, and its default is the old file.

## 3. Deploy from a clean MAIN checkout, not a feature worktree

`deploy.sh` refuses off-main (`FAIL: deploy.sh must run from the main branch.`). Run it from the local main checkout and name the service — a bare target is a fleet-wide drain:

```bash
cd ~/Projects/serverAppConfig && ./scripts/deploy.sh dankserver --service frank-archon
```

If the remote pull aborts with `Your local changes ... would be overwritten by merge`, the remote copy is a committed blob written directly on the host. Prove it before discarding:

```bash
git -C /home/dank/serverAppConfigs hash-object <file>            # working copy
git -C /home/dank/serverAppConfigs rev-parse <sha>:<file>          # candidate commit
```

Equal -> `git checkout -- <file>` is lossless (the blob is in the object store), then re-deploy. Unequal -> archive to `.AGENTTRASH/<date>-<topic>/` first, then port into the repo.

## 4. Launch the chain detached, record the pid

```bash
ssh dankserver 'cd /home/dank/campaign/qwen38-w5nl64 && \
  nohup bash run-chained.sh >> run-chained.log 2>&1 < /dev/null & \
  echo "chain_pid=$!" > run-chained.pid'
```

- `run-chained.sh` is a server-side campaign artifact (not tracked in the repo); it chains the phases and must export `FRANK_QWEN38_KLD_CORPUS` for the re-chunked file.
- Move a prior `kld-ref/` into `.AGENTTRASH/<date>-qwen38-kldref-attemptN/` first, or the new recording's artefacts mix with the old ones.
- The chain writes `/home/dank/logs/qwen38-bf16-run.rc` as its completion sentinel; watch the pid, don't poll the log.

## 5. Reading state during the outage (three traps)

- **Container storage is split**: `frank-archon` is a root unit -> its container appears only under `sudo podman ps`; the media stack (romm, copyparty, exporters) is rootless -> only plain `podman ps`. Each view legitimately shows "nothing" for the other half. `pgrep -af radiance` settles which runtime is live.
- **`frank-archon` reads `failed` while the rig holds the card.** The rig stops the unit; the container exits status=1 on stop, so systemd marks `failed` until the phase restores it. Confirm with `journalctl -u frank-archon -n 6` — `Stopping...` then `Failed with result 'exit-code'` means the outage, not a crash. `systemctl show -p ActiveEnterTimestamp` dates it.
- **Engine stdout lives in `/home/dank/logs/`, not the journal** (for halogen); for radiance read the chain log.