---
name: incident-window-reconstruction
description: "Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone."
---

# Incident window reconstruction

Use when the question is **"what was running when it broke?"** — an incident,
outage, resource blow-up, or unexplained service death — and the answer must
come from artifacts, not recollection. Typical triggers: a host went down during
an experiment, a service died mid-run, someone asks for the parameters in effect
at the moment of failure.

## Principle

Three independent clocks, cross-checked — never recollection:

1. **Per-run duration** — every script wraps commands in `time`, so the log
   holds `real 4m19.129s`.
2. **Log mtime** — the last write of that log.
3. **Journal first-failure instant** — the first error line from the affected
   service.

`start ≈ mtime − real` for the *last* run in a log, and each run nests inside
the previous one's window. The run whose `[start, end]` contains the journal's
first-failure instant is the answer.

## Collect

```bash
# 1. ordering and clocks of every artifact (log mtimes)
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'

# 2. durations + per-run section markers + the exact command lines
ssh <host> 'grep -ah "^real" <logdir>/*.log'
ssh <host> 'for f in <names>; do echo "## $f"; \
  grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
  | sed "s/\(.\{130\}\)/\1\n        /g"; \
  grep -aE "^real|_EXIT=" <logdir>/$f.log; done'

# 3. first failure of the affected service, and what preceded it
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'
```

## Traps (all hit for real)

- **`/tmp` does not survive a reboot.** Launch experiment scripts from a
  persistent path (e.g. `~/models/...`) and log beside them; if the repair path
  is a reboot (common), `/tmp` scripts *and* any sampler output written there are
  gone. On the Dank fleet: `/home/dank/models/logs/<tag>.log` survived while
  `/tmp/x4*.sh` did not.
- **A log's mtime is not proof the run finished.** A log truncated (or rotated)
  while a script still holds the fd keeps growing from that point; a 6-line log
  with an `ALL_DONE` marker means the earlier bulk was removed, not that nothing
  ran.
- **Long command lines are display-truncated by agent tools.** Wrap before
  reading: `sed 's/\(.\{130\}\)/\1\n        /g'`. Otherwise you will report
  parameters that were never used.
- **Off-by-one file names silently skip evidence.** `x4b.log` vs
  `x4b-experiment.log` in a `for f in …; do [ -f $f.log ]` loop made half the
  artifacts invisible. List the directory and match exactly.
- **Overlap is a hypothesis, not an observation.** Two resource-heavy
  invocations in the same minute may or may not have been resident at once;
  say so explicitly unless a sampler or cgroup counter proves it.
- **Do not invent the cause.** Reconstruct the *window* (evidence) and keep the
  *mechanism* separate: grep the proof that would establish it (`journalctl -k |
  grep -i oom` for a killed process, driver logs for a GPU fault). If it is not
  there, publish "trigger not established" — an invented cause is worse than an
  open question because the next agent plans against it.

## Fleet-specific sensitivity

On this fleet, container healthchecks shell out to the **`podman` CLI**, so the
first `invalid internal status … could not find any running process` in the user
journal is an *early, per-minute* indicator of rootless-podman state death — and
it timestamps the incident to within one healthcheck interval. That is how the
18:58:30 Dankclaw break was pinned to the bench still loading at the time.

## Worked example (2026-09-16, Dankclaw)

| evidence | value |
|---|---|
| log mtimes | x4e-run 18:58:29, x4f-run 18:59:56 |
| durations | E1 `real 4m53.946s` (llama-cli spam), F `real 5m24.785s` (exit 143 = killed) |
| journal | first `invalid internal status` 18:58:30 |
| ⇒ in flight | **F**, launched 18:54:31 (inside its 18:54:31–18:59:56 window) |

Answer delivered: the exact flags of F, plus the note that the preceding
`llama-cli` had written 3.19 GB of spinner lines and that the `/tmp` VRAM sampler
was gone — so the overlap of the two 111 GiB-class invocations stayed marked as
[INFERENCE] rather than becoming a claim.