Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
93 lines
4.5 KiB
Markdown
93 lines
4.5 KiB
Markdown
---
|
||
name: incident-window-reconstruction
|
||
description: "Reconstruct which command/config was running at the instant an incident symptom first appeared, from on-disk artifacts (log mtimes + per-run durations + journal first-failure time) when memory and /tmp are gone."
|
||
---
|
||
|
||
# Incident window reconstruction
|
||
|
||
Use when the question is **"what was running when it broke?"** — an incident,
|
||
outage, resource blow-up, or unexplained service death — and the answer must
|
||
come from artifacts, not recollection. Typical triggers: a host went down during
|
||
an experiment, a service died mid-run, someone asks for the parameters in effect
|
||
at the moment of failure.
|
||
|
||
## Principle
|
||
|
||
Three independent clocks, cross-checked — never recollection:
|
||
|
||
1. **Per-run duration** — every script wraps commands in `time`, so the log
|
||
holds `real 4m19.129s`.
|
||
2. **Log mtime** — the last write of that log.
|
||
3. **Journal first-failure instant** — the first error line from the affected
|
||
service.
|
||
|
||
`start ≈ mtime − real` for the *last* run in a log, and each run nests inside
|
||
the previous one's window. The run whose `[start, end]` contains the journal's
|
||
first-failure instant is the answer.
|
||
|
||
## Collect
|
||
|
||
```bash
|
||
# 1. ordering and clocks of every artifact (log mtimes)
|
||
ssh <host> 'ls -l --time-style=+%m-%d_%H:%M:%S <logdir>/*.log | awk "{print \$6, \$7}"'
|
||
|
||
# 2. durations + per-run section markers + the exact command lines
|
||
ssh <host> 'grep -ah "^real" <logdir>/*.log'
|
||
ssh <host> 'for f in <names>; do echo "## $f"; \
|
||
grep -aE "^####|^\+ /path/to/binary" <logdir>/$f.log \
|
||
| sed "s/\(.\{130\}\)/\1\n /g"; \
|
||
grep -aE "^real|_EXIT=" <logdir>/$f.log; done'
|
||
|
||
# 3. first failure of the affected service, and what preceded it
|
||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -a "invalid internal status" | head -2'
|
||
ssh <host> 'journalctl --user --since "<date> <HH:MM>" | grep -aiE "Started|Created container" | tail -6'
|
||
```
|
||
|
||
## Traps (all hit for real)
|
||
|
||
- **`/tmp` does not survive a reboot.** Launch experiment scripts from a
|
||
persistent path (e.g. `~/models/...`) and log beside them; if the repair path
|
||
is a reboot (common), `/tmp` scripts *and* any sampler output written there are
|
||
gone. On the Dank fleet: `/home/dank/models/logs/<tag>.log` survived while
|
||
`/tmp/x4*.sh` did not.
|
||
- **A log's mtime is not proof the run finished.** A log truncated (or rotated)
|
||
while a script still holds the fd keeps growing from that point; a 6-line log
|
||
with an `ALL_DONE` marker means the earlier bulk was removed, not that nothing
|
||
ran.
|
||
- **Long command lines are display-truncated by agent tools.** Wrap before
|
||
reading: `sed 's/\(.\{130\}\)/\1\n /g'`. Otherwise you will report
|
||
parameters that were never used.
|
||
- **Off-by-one file names silently skip evidence.** `x4b.log` vs
|
||
`x4b-experiment.log` in a `for f in …; do [ -f $f.log ]` loop made half the
|
||
artifacts invisible. List the directory and match exactly.
|
||
- **Overlap is a hypothesis, not an observation.** Two resource-heavy
|
||
invocations in the same minute may or may not have been resident at once;
|
||
say so explicitly unless a sampler or cgroup counter proves it.
|
||
- **Do not invent the cause.** Reconstruct the *window* (evidence) and keep the
|
||
*mechanism* separate: grep the proof that would establish it (`journalctl -k |
|
||
grep -i oom` for a killed process, driver logs for a GPU fault). If it is not
|
||
there, publish "trigger not established" — an invented cause is worse than an
|
||
open question because the next agent plans against it.
|
||
|
||
## Fleet-specific sensitivity
|
||
|
||
On this fleet, container healthchecks shell out to the **`podman` CLI**, so the
|
||
first `invalid internal status … could not find any running process` in the user
|
||
journal is an *early, per-minute* indicator of rootless-podman state death — and
|
||
it timestamps the incident to within one healthcheck interval. That is how the
|
||
18:58:30 Dankclaw break was pinned to the bench still loading at the time.
|
||
|
||
## Worked example (2026-09-16, Dankclaw)
|
||
|
||
| evidence | value |
|
||
|---|---|
|
||
| log mtimes | x4e-run 18:58:29, x4f-run 18:59:56 |
|
||
| durations | E1 `real 4m53.946s` (llama-cli spam), F `real 5m24.785s` (exit 143 = killed) |
|
||
| journal | first `invalid internal status` 18:58:30 |
|
||
| ⇒ in flight | **F**, launched 18:54:31 (inside its 18:54:31–18:59:56 window) |
|
||
|
||
Answer delivered: the exact flags of F, plus the note that the preceding
|
||
`llama-cli` had written 3.19 GB of spinner lines and that the `/tmp` VRAM sampler
|
||
was gone — so the overlap of the two 111 GiB-class invocations stayed marked as
|
||
[INFERENCE] rather than becoming a claim.
|