43 lines
3.7 KiB
Markdown
43 lines
3.7 KiB
Markdown
---
|
||
name: bench-arm-provenance-recovery
|
||
description: "Before calling a leftover LLM bench drop-in a rogue/hijack, reconstruct whether Frank ordered and measured that arm: omp archive prompt timestamps + journald print_timing for the exact pid, then land the verdict in the repo."
|
||
---
|
||
|
||
Use when a live service boots a payload you did not configure (stale `systemd` drop-in / `*_ENV_FILE` arm env) and the docs — or your own compaction-inherited context — call it "unfinished", "armed but never measured", or "hijack". Do NOT report that until proven: the arm may be an ordered, measured, approved configuration whose record lived only in `/tmp` + journald.
|
||
|
||
## Why this exists
|
||
2026-10-18 `frank-zealot`: a `frank-zealot.service.d/ptq1.conf` drop-in made the cutover deploy boot a different model at a different shape. I reported it as an unfinished experiment. It was actually the 27B PTQ1_0 arm Frank ordered at 02:36, that served production 03:14:39→03:38:57, and whose numbers existed only in tool output that context compaction later dropped.
|
||
|
||
## Step 1 — recover the ORDER (the harness archive, not memory)
|
||
`eval` with the `archive` prelude:
|
||
|
||
```python
|
||
ps = await archive.prompts() # searchable prompt history, survives compaction
|
||
# filter for the payload name / repo URL keywords, print p['at'] + p['text'][:90]
|
||
r = await archive.session("<sessionId>") # full session record
|
||
recaps = await archive.recaps() # recap stamps — check they postdate the window
|
||
```
|
||
|
||
`archive.prompts()` keeps **user prompts with ISO timestamps** even when the middle of the transcript was dropped. Timestamp every instruction: ordered → measured → superseded. Convert to local (`UTC−7` here) before quoting. A recap older than the instruction proves the recap layer never covered the window.
|
||
|
||
## Step 2 — recover the MEASUREMENT (journal, per pid)
|
||
The engine's own per-request timings are the record; they outlive `/tmp` env files.
|
||
|
||
```bash
|
||
ssh dank@<host> 'journalctl -u <svc> --no-pager' | grep -F 'serve-<svc>.sh[<PID>]' \
|
||
| grep -E 'prompt eval time|eval time' | awk '... median per task ...'
|
||
```
|
||
|
||
- Pin the **pid** (`sh[NNNN]`): `--since` windows alone pull the previous process's numbers too.
|
||
- llama.cpp format: `prompt eval time = 1234.56 ms / 100 tokens (…, 81.03 tokens per second)`; decode rows are `eval time = … ms / N tokens (…)`. Multiple spaces between `task`/`eval` break naive regexes — match `task\s+(\d+)\s+prompt eval time` exactly, or parse both labels separately.
|
||
- Filter pathological rows: generations running concurrently on other slots, or ending in a 4-byte EOS, read ~4× low. Report clean single-stream probes (≥200 generated tokens) and label the rest.
|
||
- Cross-check the boot banner for the payload identity (`type ptq1_0: 402 tensors`, KV ledger, slot/ctx lines) — that proves *which* shape that pid served.
|
||
|
||
## Step 3 — adjudicate, then land it in the repo
|
||
Write the verdict in the same session you derive it: runbook deploy-ledger section, `log.md` dated section, the arm script's header comment, and the AGENTS.md law. `/tmp` and journald are not records — that absence is what makes the next agent misread an approved arm as a rogue one.
|
||
|
||
## Traps
|
||
- `mask` fails on a relinked unit; never `systemctl disable`; retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload` (recoverable, not deleted).
|
||
- `/health` 200, `/v1/models` echoing the alias, and a green e2e gate all fail to read the payload — check `systemctl show <svc> -p DropInPaths` (must be empty), then `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger.
|
||
- Verification sequence that is honest: correction commit must quote commands you ran; state `[INFERENCE]` for anything the journal does not show.
|