command-center/agent/managed-skills/incident-window-reconstruction/SKILL.md

name, description

name	description
incident-window-reconstruction	Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone.

Incident window reconstruction

Use when the question is "what was running when it broke?" — an incident, outage, resource blow-up, or unexplained service death — and the answer must come from artifacts, not recollection. Typical triggers: a host went down during an experiment, a service died mid-run, someone asks for the parameters in effect at the moment of failure.

Principle

Three independent clocks, cross-checked — never recollection:

1. Per-run duration — every script wraps commands in time, so the log holds real 4m19.129s.
2. Log mtime — the last write of that log.
3. Journal first-failure instant — the first error line from the affected service.

start ≈ mtime − real for the last run in a log, and each run nests inside the previous one's window. The run whose [start, end] contains the journal's first-failure instant is the answer.

Collect

# 1. ordering and clocks of every artifact (log mtimes)
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'

# 2. durations + per-run section markers + the exact command lines
ssh <host> 'grep -ah "^real" <logdir>/*.log'
ssh <host> 'for f in <names>; do echo "## $f"; \
  grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
  | sed "s/\(.\{130\}\)/\1\n        /g"; \
  grep -aE "^real|_EXIT=" <logdir>/$f.log; done'

# 3. first failure of the affected service, and what preceded it
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'

Traps (all hit for real)

• /tmp does not survive a reboot. Launch experiment scripts from a persistent path (e.g. ~/models/...) and log beside them; if the repair path is a reboot (common), /tmp scripts and any sampler output written there are gone. On the Dank fleet: /home/dank/models/logs/<tag>.log survived while /tmp/x4*.sh did not.
• A log's mtime is not proof the run finished. A log truncated (or rotated) while a script still holds the fd keeps growing from that point; a 6-line log with an ALL_DONE marker means the earlier bulk was removed, not that nothing ran.
• Long command lines are display-truncated by agent tools. Wrap before reading: sed 's/\(.\{130\}\)/\1\n /g'. Otherwise you will report parameters that were never used.
• Off-by-one file names silently skip evidence. x4b.log vs x4b-experiment.log in a for f in …; do [ -f $f.log ] loop made half the artifacts invisible. List the directory and match exactly.
• Overlap is a hypothesis, not an observation. Two resource-heavy invocations in the same minute may or may not have been resident at once; say so explicitly unless a sampler or cgroup counter proves it.
• Do not invent the cause. Reconstruct the window (evidence) and keep the mechanism separate: grep the proof that would establish it (journalctl -k | grep -i oom for a killed process, driver logs for a GPU fault). If it is not there, publish "trigger not established" — an invented cause is worse than an open question because the next agent plans against it.

Fleet-specific sensitivity

On this fleet, container healthchecks shell out to the podman CLI, so the first invalid internal status … could not find any running process in the user journal is an early, per-minute indicator of rootless-podman state death — and it timestamps the incident to within one healthcheck interval. That is how the 18:58:30 Dankclaw break was pinned to the bench still loading at the time.

Worked example (2026-09-16, Dankclaw)

evidence	value
log mtimes	x4e-run 18:58:29, x4f-run 18:59:56
durations	E1 real 4m53.946s (llama-cli spam), F real 5m24.785s (exit 143 = killed)
journal	first invalid internal status 18:58:30
⇒ in flight	F, launched 18:54:31 (inside its 18:54:31–18:59:56 window)

Answer delivered: the exact flags of F, plus the note that the preceding llama-cli had written 3.19 GB of spinner lines and that the /tmp VRAM sampler was gone — so the overlap of the two 111 GiB-class invocations stayed marked as [INFERENCE] rather than becoming a claim.