command-center/agent/managed-skills/qwen38-chain-rerun-dankserver/SKILL.md

name, description

name	description
qwen38-chain-rerun-dankserver	Rerun the qwen38-w5nl64 campaign on dankserver: re-chunk the KLD corpus to the real KV pool, verify the rig's gate with --dry-run, deploy, and relaunch run-chained.sh detached — plus the three state-reading traps (root vs rootless podman, unit failed during an outage, the .rc sentinel).

Qwen38 campaign rerun on dankserver

Use when rerunning the qwen38-w5nl64 campaign chain (kldref -> calib -> retire the reference .rad) on Dankserver, or when the chain died with E kld: document N is <n> tokens and the KV pool holds <pool> and needs its corpus fixed.

Order is fixed: corpus -> gate -> deploy -> chain. Skipping the gate burns the outage window.

1. Corpus must fit the pool, not the flag

The rig's --max-model-len is decorative: the recorder refuses any document longer than the KV pool the boot actually got (printed by the boot ledger, e.g. 7,436), so a corpus authored at a wider window scores nothing.

ssh dankserver '/home/dank/venvs/radcorpus/bin/python \
  /home/dank/campaign/qwen38-w5nl64/rechunk-kld-corpus.py \
  --corpus /home/dank/corpora/out/kld.jsonl \
  --out    /home/dank/corpora/out/kld-7168.jsonl'

Read the summary, don't trust the run: tokens_in must equal tokens_out (that equality is the whole invariant — the tool exists to prove it), longest_document must equal the ceiling, and scored_positions grows because every window now carries its own score_from.

Chars-per-token is payload-specific: this file is chars/3.6, another corpus measured chars/4.97. Never size a window from chars alone.

2. Gate the rig before spending the outage

scripts/run-qwen38-bf16.sh kldref --dry-run runs the corpus gate and prints the argv without touching the production unit:

ssh dankserver 'cd /home/dank/serverAppConfigs && \
  FRANK_QWEN38_KLD_CORPUS=/home/dank/corpora/out/kld-7168.jsonl \
  bash scripts/run-qwen38-bf16.sh kldref --dry-run'

Check the printed line corpus fits: longest record <chars> ... inside the <window> and that the -v ...:/data/kld.jsonl:ro mount names the file you intended — the rig mounts whatever FRANK_QWEN38_KLD_CORPUS points at, and its default is the old file.

3. Deploy from a clean MAIN checkout, not a feature worktree

deploy.sh refuses off-main (FAIL: deploy.sh must run from the main branch.). Run it from the local main checkout and name the service — a bare target is a fleet-wide drain:

cd ~/Projects/serverAppConfig && ./scripts/deploy.sh dankserver --service frank-archon

If the remote pull aborts with Your local changes ... would be overwritten by merge, the remote copy is a committed blob written directly on the host. Prove it before discarding:

git -C /home/dank/serverAppConfigs hash-object <file>            # working copy
git -C /home/dank/serverAppConfigs rev-parse <sha>:<file>          # candidate commit

Equal -> git checkout -- <file> is lossless (the blob is in the object store), then re-deploy. Unequal -> archive to .AGENTTRASH/<date>-<topic>/ first, then port into the repo.

4. Launch the chain detached, record the pid

ssh dankserver 'cd /home/dank/campaign/qwen38-w5nl64 && \
  nohup bash run-chained.sh >> run-chained.log 2>&1 < /dev/null & \
  echo "chain_pid=$!" > run-chained.pid'

• run-chained.sh is a server-side campaign artifact (not tracked in the repo); it chains the phases and must export FRANK_QWEN38_KLD_CORPUS for the re-chunked file.
• Move a prior kld-ref/ into .AGENTTRASH/<date>-qwen38-kldref-attemptN/ first, or the new recording's artefacts mix with the old ones.
• The chain writes /home/dank/logs/qwen38-bf16-run.rc as its completion sentinel; watch the pid, don't poll the log.

5. Reading state during the outage (three traps)

• Container storage is split: frank-archon is a root unit -> its container appears only under sudo podman ps; the media stack (romm, copyparty, exporters) is rootless -> only plain podman ps. Each view legitimately shows "nothing" for the other half. pgrep -af radiance settles which runtime is live.
• frank-archon reads failed while the rig holds the card. The rig stops the unit; the container exits status=1 on stop, so systemd marks failed until the phase restores it. Confirm with journalctl -u frank-archon -n 6 — Stopping... then Failed with result 'exit-code' means the outage, not a crash. systemctl show -p ActiveEnterTimestamp dates it.
• Engine stdout lives in /home/dank/logs/, not the journal (for halogen); for radiance read the chain log.