command-center/agent/managed-skills/dankclaw-boot-line-window/SKILL.md

name, description

name	description
dankclaw-boot-line-window	Change a Dankclaw kernel command-line parameter safely: tracked grub + installer, measured before/after A/B on live LLM traffic, post-boot gates.

Dank fleet kernel boot-line window (Dankclaw pattern)

Kernel command-line state is host state the deployer never owns. The repo law ("all edits through git") still applies: fetch the live file, commit it, edit the tracked copy, apply with a script.

Source of truth

• dankclaw/host/grub — tracked copy of /etc/default/grub.
• scripts/set-dankclaw-bootline.sh — the only door: diff → sudo install -m 0644 -o root -g root → sudo update-grub → print the default grub.cfg entry → reboot only on --reboot. Refuses any host whose hostname -s lacks dankclaw.
• Decision record: serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md.
• deploy-plan.py maps dankclaw/* subdirs explicitly; a new dankclaw/host/ path plans nothing (verify with a --dry-run).

Procedure

1. Capture the before-arm measurement first — you cannot get it back after the reboot.
2. Fetch provenance byte-exact: ssh dankclaw 'sudo cat /etc/default/grub' > dankclaw/host/grub (never retype; a transcription dropped a leading 0).
3. Commit provenance, edit the tracked copy, commit the flip, push (Gitea lives on Dankclaw — pull on the host before rebooting), then ssh dankclaw 'cd ~/serverAppConfigs && git pull --ff-only origin main'.
4. Install without --reboot first: it prints the grub.cfg default entry; confirm the token there before the irreversible step.
5. ssh dankclaw 'sudo systemctl reboot --no-block'.
6. Poll on /proc/uptime + cmdline, never on ping — the NIC answers ICMP for a minute after shutdown and sshd reset looks like "reachable". Gate set: systemctl --failed count 0 · systemctl is-system-running running · /proc/cmdline token · ls /sys/kernel/iommu_groups | wc -l · cat /sys/kernel/iommu_groups/*/type | sort | uniq -c · systemctl is-active frank-dragoon frank-zealot · curl 127.0.0.1:7770/health (listener binds only after the full load — poll up to 10 min).

Measurement discipline (this is where windows get lied about)

• Both arms must be byte-identical traffic. Engine-reported token counts are the identity proof (e.g. 59,024 prefill / 21,568 needle both boots).
• /tmp is tmpfs: the reboot deletes /tmp/halogen-eval. Rebuild with scripts/stage-frank-dragoon-eval.sh (corpus 2,891 lines / 1,287,656 B is the check) — and run it via bash -lc, uv is in ~/.local/bin, absent from a non-login PATH.
• The 51.2 GB halogen n-gram table is page cache (Pin::None). A fresh boot has it cold and a cold table throttles prefill → cat it once (~40 s) before measuring. WARM=1 in the staging script does this.
• Sampled decode on this engine is a useless single-sample instrument: three repeats of the same request gave 31.8 / 46.9 / 47.8 t/s. Only prefill (cold) and needle-hit carry signal; anything else needs ≥3 repeats.
• Bench law still bites: nothing else on the GPU; measuring through the live service is how dragoon/zealot are measured (a scratch server cannot share the card).

Traps

• edit PUT N.: can insert instead of replace — verify the range shrank, then CUT the leftover.
• Body rows beginning with a markdown bullet (- ) get auto-prefixed and land as +- ; rewrite the block with write when that happens.
• A long reboot fires LlmEndpointDown → Signal; Alertmanager has no persistence, so every active alert gets a fresh startsAt after boot — check query_range history before blaming the window for an alert.
• sudo podman ps shows nothing for Dankclaw's rootless stack; systemctl is-active caddy is meaningless for quadlets. Use the public endpoints or systemctl --user as dank.

Worked example (2026-10-18)

amd_iommu=off → iommu=pt to restore the XDNA2 NPU. Result: cold prefill 1,585.1 → 1,616.0 t/s (+2.0 %, i.e. no penalty), needle hit/hit, amdxdna bound 0000:c6:00.1 (group 26 identity, PASID width 10). The 13–16 % figure it replaced had never been measured locally. Deferred on purpose: pp_dpm_fclk pin (nothing co-runs yet) and PerfOpt (needs Linux 7.4; Ubuntu 26.04 caps at 7.0.0-38.38).