44 lines
4.2 KiB
Markdown
44 lines
4.2 KiB
Markdown
---
|
||
name: dankclaw-boot-line-window
|
||
description: "Change a Dankclaw kernel command-line parameter safely: tracked grub + installer, measured before/after A/B on live LLM traffic, post-boot gates."
|
||
---
|
||
|
||
# Dank fleet kernel boot-line window (Dankclaw pattern)
|
||
|
||
Kernel command-line state is **host state the deployer never owns**. The repo law
|
||
("all edits through git") still applies: fetch the live file, commit it, edit the
|
||
tracked copy, apply with a script.
|
||
|
||
## Source of truth
|
||
- `dankclaw/host/grub` — tracked copy of `/etc/default/grub`.
|
||
- `scripts/set-dankclaw-bootline.sh` — the only door: diff → `sudo install -m 0644 -o root -g root` →
|
||
`sudo update-grub` → print the **default `grub.cfg` entry** → reboot only on `--reboot`.
|
||
Refuses any host whose `hostname -s` lacks `dankclaw`.
|
||
- Decision record: `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
||
- `deploy-plan.py` maps `dankclaw/*` subdirs explicitly; a new `dankclaw/host/` path plans nothing (verify with a `--dry-run`).
|
||
|
||
## Procedure
|
||
1. **Capture the before-arm measurement first** — you cannot get it back after the reboot.
|
||
2. Fetch provenance byte-exact: `ssh dankclaw 'sudo cat /etc/default/grub' > dankclaw/host/grub` (never retype; a transcription dropped a leading `0`).
|
||
3. Commit provenance, edit the tracked copy, commit the flip, push (**Gitea lives on Dankclaw — pull on the host before rebooting**), then `ssh dankclaw 'cd ~/serverAppConfigs && git pull --ff-only origin main'`.
|
||
4. Install **without** `--reboot` first: it prints the `grub.cfg` default entry; confirm the token there before the irreversible step.
|
||
5. `ssh dankclaw 'sudo systemctl reboot --no-block'`.
|
||
6. **Poll on `/proc/uptime` + cmdline, never on ping** — the NIC answers ICMP for a minute after shutdown and sshd reset looks like "reachable".
|
||
Gate set: `systemctl --failed` count 0 · `systemctl is-system-running` running · `/proc/cmdline` token · `ls /sys/kernel/iommu_groups | wc -l` · `cat /sys/kernel/iommu_groups/*/type | sort | uniq -c` · `systemctl is-active frank-dragoon frank-zealot` · `curl 127.0.0.1:7770/health` (listener binds only after the full load — poll up to 10 min).
|
||
|
||
## Measurement discipline (this is where windows get lied about)
|
||
- Both arms must be **byte-identical traffic**. Engine-reported token counts are the identity proof (e.g. 59,024 prefill / 21,568 needle both boots).
|
||
- `/tmp` is **tmpfs**: the reboot deletes `/tmp/halogen-eval`. Rebuild with `scripts/stage-frank-dragoon-eval.sh` (corpus 2,891 lines / 1,287,656 B is the check) — and run it via `bash -lc`, `uv` is in `~/.local/bin`, absent from a non-login PATH.
|
||
- The 51.2 GB halogen n-gram table is page cache (`Pin::None`). A fresh boot has it cold and a cold table throttles prefill → `cat` it once (~40 s) before measuring. `WARM=1` in the staging script does this.
|
||
- **Sampled decode on this engine is a useless single-sample instrument**: three repeats of the same request gave 31.8 / 46.9 / 47.8 t/s. Only prefill (cold) and needle-hit carry signal; anything else needs ≥3 repeats.
|
||
- Bench law still bites: nothing else on the GPU; measuring *through* the live service is how dragoon/zealot are measured (a scratch server cannot share the card).
|
||
|
||
## Traps
|
||
- `edit` `PUT N.:` can **insert instead of replace** — verify the range shrank, then `CUT` the leftover.
|
||
- Body rows beginning with a markdown bullet (`- `) get auto-prefixed and land as `+- `; rewrite the block with `write` when that happens.
|
||
- A long reboot fires `LlmEndpointDown` → Signal; Alertmanager has **no persistence**, so every active alert gets a fresh `startsAt` after boot — check `query_range` history before blaming the window for an alert.
|
||
- `sudo podman ps` shows nothing for Dankclaw's rootless stack; `systemctl is-active caddy` is meaningless for quadlets. Use the public endpoints or `systemctl --user` as `dank`.
|
||
|
||
## Worked example (2026-10-18)
|
||
`amd_iommu=off` → `iommu=pt` to restore the XDNA2 NPU. Result: cold prefill 1,585.1 → 1,616.0 t/s (**+2.0 %**, i.e. no penalty), needle hit/hit, `amdxdna` bound `0000:c6:00.1` (group 26 identity, PASID width 10). The 13–16 % figure it replaced had never been measured locally. Deferred on purpose: `pp_dpm_fclk` pin (nothing co-runs yet) and PerfOpt (needs Linux 7.4; Ubuntu 26.04 caps at `7.0.0-38.38`).
|