65 lines
3.4 KiB
Markdown
65 lines
3.4 KiB
Markdown
---
|
||
name: self-severing-model-swap-cutover
|
||
description: "Swap the model on frank-archon (the endpoint the operator session runs on): full pre-cutover repo unit, detached deploy that survives session death, post-return boot-ledger harvest, e2e attribution."
|
||
---
|
||
|
||
# Swapping the model you are running on (frank-archon)
|
||
|
||
The harness agent shares the endpoint being swapped: the deploy restarts
|
||
frank-archon, the session dies mid-tool-call, the harness catches the return.
|
||
Plan for the severing; never keep critical state only in-session.
|
||
|
||
## Pre-cutover (all must be committed/pushed BEFORE the deploy)
|
||
1. Read `scripts/frank-archon.env` live state and record the shape; the swap
|
||
changes ONLY `MODEL_FILE` (fleet model-swap law) unless asked otherwise.
|
||
2. Fetch script per payload (hfget + exact bytes + Hub LFS sha256 +
|
||
`--verify-only`, no token on the command line); run detached via
|
||
`systemd-run --unit=<name>-fetch` on dankserver.
|
||
3. Re-key in one commit: `frank-archon.env` (+ trio headers),
|
||
`tests/frank_archon_unit.bats` payload id/sha256 (keep the outgoing
|
||
payload as the documented rollback line), homepage card
|
||
(`shared/homepage/services.json`), runbook (new era page + index flips +
|
||
old page retired), `AGENTS.md` topology row/section/runbook index.
|
||
4. Full `./tests/run.sh` green, push origin main, `git pull --ff-only` on
|
||
dankserver. Verify the engine pin's flag set matches the payload card
|
||
(radiance pins have drifted on flag renames).
|
||
5. Disk: check `df -h /home/dank` before launching a >100 GB download; the
|
||
outgoing payload stays on disk as rollback — deleting it requires Frank's
|
||
one-time `rm` authorization.
|
||
|
||
## Cutover, detached from the session
|
||
Launch from the workstation as a transient unit so ssh/session teardown
|
||
cannot kill it:
|
||
```
|
||
setsid systemd-run --unit=archon-<tag>-cutover --remain-after-exit \
|
||
bash -lc 'cd <repo> && ./scripts/deploy.sh dankserver --service frank-archon \
|
||
> ~/archon-<tag>-cutover.log 2>&1; echo $? > ~/archon-<tag>.rc'
|
||
```
|
||
The session dies when frank-archon restarts (~1–2 min load). On return:
|
||
read the `.rc` sentinel and log tail; then the journal ledger:
|
||
```
|
||
journalctl -u frank-archon --no-pager --since "<deploy>" | grep -iE "kv cache|slab|sampler|MTP|prefix"
|
||
```
|
||
|
||
## Post-boot harvest (fill the runbook placeholders)
|
||
- KV: `kv_attn` KiB/4-token block, pool token count vs `seqs × max-model-len`.
|
||
- Expert slab slots per rank (compare to the outgoing payload).
|
||
- Sampler rows = `seqs × (1 + spec)` (wht6 exactness bound 25).
|
||
- Prefix-cache RECOVERED line (survives swaps when arch+tokenizer unchanged).
|
||
- Vision probe: solid-color PNG ≥256 px, read `reasoning_content` (small
|
||
max_tokens → thinking eats the budget, `content` empty with
|
||
`finish_reason: stop` = healthy, not broken).
|
||
- Decode ladder needs a stop-and-mask window — separate bench task.
|
||
|
||
## e2e gate redness on dankclaw surfaces
|
||
`deploy.sh dankserver` runs the Playwright gate against the LIVE fleet, so
|
||
dankclaw faults fail your deploy. Attribute before believing: crawl 502 /
|
||
SSO fail / Marreta tile → run the
|
||
`gluetun-dependent-boot-race-recovery` census first (reboot-triggered,
|
||
unrelated to the swap), cascade-restart, rerun `tests/e2e/run.sh` directly —
|
||
a frank-archon health gate that passed means the swap itself is fine.
|
||
|
||
## Rollback
|
||
Revert the swap commit, redeploy `--service frank-archon` — the outgoing
|
||
`.rad` remains on disk (until deleted by authorized `rm`).
|