live: heartbeat sync 2026-10-30T21:32:40-07:00 (Refs #11)
This commit is contained in:
Frank committed
1 parent
d03fe57bbd
commit
d627368f71
13 files changed
+280
-26
No files matched your search
@@ -1,9 +1,9 @@
|
||||
---
|
||||
name: agenttrash-purger-invocation
|
||||
description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) — plus its two refusal traps: ENOSPC masquerading as a stale lock, and an unreadable container-storage directory poisoning the whole root scan."
|
||||
description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) — plus its refusal traps (ENOSPC as a fake stale lock, unreadable dirs poisoning a root) and the per-root scope trap."
|
||||
---
|
||||
|
||||
# Purging .AGENTTRASH — invocation, and the two traps that make it refuse
|
||||
# Purging .AGENTTRASH — invocation, and the traps that make it refuse or over-reach
|
||||
|
||||
`scripts/purge-agenttrash.sh` (serverAppConfigs, merged in PR #79 `fad87aa`) is the tool form of the fleet's deletion policy: files land in `.AGENTTRASH`, and an approved purge runs this instead of being improvised per session.
|
||||
|
||||
@@ -19,6 +19,10 @@ sh scripts/purge-agenttrash.sh --execute /some/root # scope to one root (defaul
|
||||
- Every printable line is fixed at plan time in `$HOME/.agenttrash-purge.lock/plan`, so dry-run and `--execute` output cannot drift. Exit `0` ok · `1` any refusal or failure · `2` usage.
|
||||
- Deleting a big staged item by hand (e.g. `find <dir> -depth -delete`) is the fallback **only** when the tool cannot run at all — see trap 1 — and the operator's purge approval covers exactly that.
|
||||
|
||||
## Trap 0: the approval is per item, the scope is per root
|
||||
|
||||
The plan verb `empty <root>/.AGENTTRASH` means **the whole trash directory**, not the one item the operator approved. So before `--execute`, list the root and prove it holds only the approved staged item — `ls -A <root>/.AGENTTRASH && du -sh <root>/.AGENTTRASH/*` — and note that a second `.AGENTTRASH` deeper in the tree is also in scope (`plan remove` on it = it was empty, safe). Measured 2026-10-30 on dankserver: `/mnt/entropy/.AGENTTRASH` held exactly the approved `20261030-movies-monk` (124 files, 32 G) plus an empty `qwen38-w5nl64/.AGENTTRASH`; plan → `--execute /mnt/entropy` reported `removed=2 deleted=2 refused=0` and free space rose 187 G → 218 G.
|
||||
|
||||
## Trap 1: a full filesystem makes it lie about a lock (measured 2026-10-30)
|
||||
|
||||
With **0 bytes free**, three invocations refused with
|
||||
@@ -39,4 +43,4 @@ and no lock directory existed anywhere. The script writes its lock *and* its pla
|
||||
|
||||
## Verified good path (dankserver, 2026-10-30)
|
||||
|
||||
After freeing space by hand: `/tmp`-scoped plan → `--execute` deleted the staged item and reported `removed=1 deleted=1`; a second `--execute /mnt/entropy` cleaned that volume's root (`refused=0`). The cleanup log carries one dated line per run; read it instead of re-deriving what happened.
|
||||
After freeing space by hand: `/tmp`-scoped plan → `--execute` deleted the staged item and reported `removed=1 deleted=1`; a second `--execute /mnt/entropy` cleaned that volume's root (`refused=0`). A 32 G staged item on an 87 T volume scans + purges in ~90 s per pass (the scan, not the deletion, is the slow part). The cleanup log carries one dated line per run — both the dry run and the execute log the verb `plan`, so tell them apart by `removed=`; read it instead of re-deriving what happened.
|
||||
@@ -1,24 +1,42 @@
|
||||
---
|
||||
name: line-edit-anchor-discipline
|
||||
description: "Avoid the recurring edit-tool line-op traps: empty PUT deletes lines, PUT ranges swallowing multi-line block delimiters, and anchors landing mid-expression. Use whenever issuing CUT/PUT line edits on tagged files."
|
||||
description: "Avoid the recurring edit-tool line-op traps: empty PUT deletes lines, PUT ranges swallowing neighbours, wrong-anchored PUTs replacing the wrong line, literal markdown bullet rows arriving verbatim as \"+-\"."
|
||||
---
|
||||
|
||||
# Line-edit anchor discipline (CUT/PUT ops on tagged files)
|
||||
# Line-edit anchor discipline
|
||||
|
||||
Five self-inflicted syntax wounds in one session (franksign, 2026-10-21) all trace to three rules.
|
||||
The `edit` tool addresses lines, not intent. Three traps cost real time; all three
|
||||
fail *silently* or with a repair notice that is easy to skim past.
|
||||
|
||||
## Rules
|
||||
1. **An empty PUT body DELETES the targeted lines** (tool warns: "Interpreted an empty PUT body as deletion"). Twice it ate a whole log entry / a closing `}`. For no-ops use nothing; for deletes use explicit `CUT`.
|
||||
2. **Never PUT a range whose endpoints cut through a multi-line block.** If a `| {`-style union variant, a frontmatter `---` fence, or a `test(...)` closer straddles the range boundary, the replacement swallows the opener/closer and strands the body. Pre-condition: line N-1 and M+1 must be block-complete. Fix-after-the-fact pattern: read the broken region raw, then `PUT k.=k:` to restore the single swallowed line (e.g. the missing `---`, the orphaned ` | {`, `}` → `});`).
|
||||
3. **Search-hit reads do not authorize PUT.** A grep/folded-summary read marks lines as "never displayed"; the edit tool refuses and echoes actual content. That refusal is a free verification — re-check the echoed lines match intent, then straight-retry with the same tag.
|
||||
## 1. Empty PUT deletes the range
|
||||
|
||||
## Costly mistakes seen
|
||||
- Empty `PUT 7-7:` deleted a just-written log entry; restoring required re-issuing the full entry text.
|
||||
- `PUT 6.=7` replacing `timestamp`+`---` dropped the frontmatter close fence → lint-breaking file silently.
|
||||
- `PUT 10.=11` on a union type swallowed ` | {` opening a 12-line variant → cascade of LSP "unexpected label" warnings.
|
||||
- `CUT` of a stranded opener left its dangling `.json())` behind; balance errors surface only at EOF (`expected ')' but file ends`) — locate by reading regions raw, not by widening the payload.
|
||||
`PUT n.=m:` with no body rows deletes lines n..m. To insert, use `PUT >n:`
|
||||
(after line n) or `PUT <n:` (before line n). Never send a PUT range whose body is
|
||||
empty unless deletion is the goal.
|
||||
|
||||
## Discipline
|
||||
- After any edit that reports a syntax error or a warning: read the touched region with `:raw` before the next op; fix the single wrong line, do not resend the block.
|
||||
- "no change: byte-identical" means your anchor math is off — re-read, don't append more content.
|
||||
- Prefer replacing whole coherent blocks (start line = block start, end line = block end) over edits that end mid-statement.
|
||||
## 2. PUT ranges must be the complete final content of exactly those lines
|
||||
|
||||
- Replacing one line: `PUT n.=n:` with exactly one body row. If the body carries
|
||||
the *previous* line as its first row (a copy-paste of the anchor), that line is
|
||||
duplicated and the real target line is overwritten — no error, and the
|
||||
"auto-repaired replacement boundary" note is the only hint.
|
||||
- Anchor only on lines the current `[path#tag]` display actually showed. If the
|
||||
tool answers "anchors to lines you never displayed", the line numbers are
|
||||
wrong: re-read that range, then re-issue. Do not retry blind.
|
||||
- Multi-op edits: order ops bottom-up (highest line numbers first) so earlier
|
||||
inserts cannot shift later anchors.
|
||||
|
||||
## 3. Literal `- ` rows (markdown bullets) cannot go through `edit`
|
||||
|
||||
A body row starting with `-` parses as a CUT op. The error says to prefix it with
|
||||
`+` (`+- item`) — **that advice is wrong in practice**: the file ends up
|
||||
containing the literal `+- item`. When the content legitimately needs leading
|
||||
`- ` bullets (prompt law text, changelog entries, lists), write it with `write`
|
||||
after a full `read` of the file, or restructure the line to avoid the leading
|
||||
dash.
|
||||
|
||||
## Verification habit
|
||||
|
||||
After any PUT that replaces a range, `read` the touched range (not the whole
|
||||
file) and confirm the neighbouring lines survived — especially the line after the
|
||||
range, which is what a swallowed boundary eats. Cheap; catches the silent cases.
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
name: magic-number-provenance-audit
|
||||
description: "Audit every numeric threshold in a change before shipping it: classify provenance (inherited port / declared authority / invented), delete invented arithmetic, and replace budgets with derivations quoted from the repo's own config. Use when adding or reviewing a cap, window, timeout, ratio, or size ladder, or when asked \"what's with the magic numbers?\"."
|
||||
---
|
||||
|
||||
# Magic-number provenance audit
|
||||
|
||||
A threshold with no source is a bug with a comment on it. The deliverable is either a
|
||||
derivation from an authority the repo already trusts, or deletion of the arithmetic.
|
||||
|
||||
## 1. Census, do not defend
|
||||
|
||||
Grep the changed paths for every numeric literal (exclude the module being rewritten so
|
||||
its own digits don't hide the pattern):
|
||||
|
||||
```
|
||||
grep -rn "1536\|4000\|700\|6000\|n_ctx\|contextWindow\|maxTokens\|tokens" src/*.ts
|
||||
```
|
||||
|
||||
Table per number: value | file:line | **provenance** | verdict. Provenance is exactly one of:
|
||||
|
||||
- **inherited port** — copied from upstream, pinned by a contract test. Keep, but say so in
|
||||
the doc comment.
|
||||
- **declared authority** — a field the runtime/fleet config already publishes.
|
||||
- **derived** — computed from beats it guards (e.g. lease TTL = 3 heartbeats).
|
||||
- **invented** — yours. This is the finding. Say "mine, invented", not "chosen".
|
||||
|
||||
Also flag the *application* of a number, not just its value: a slice like
|
||||
`text.slice(0, MAX_INPUT)` is a **head-only silent drop** (loses the tail of a dump);
|
||||
a floor is a different animal from a cap even at the same digit.
|
||||
|
||||
## 2. Find the authority before writing a digit
|
||||
|
||||
Rank of acceptable sources, best first:
|
||||
|
||||
1. **The runtime's own published field.** `~/.omp/agent/models.yml` model rows carry
|
||||
`contextWindow` (PER-SLOT for llama.cpp: it divides `--ctx-size` across slots) and
|
||||
`maxTokens`. The fleet's own clamp law is quoted in the comments there — reuse its shape:
|
||||
`chars = (contextWindow - maxTokens - 4096) * 4 - lawChars` (`4` = the file's `chars/4`
|
||||
token estimator, `4096` = its clamp margin). Then every figure in the derivation is a
|
||||
quotation, not a guess.
|
||||
2. **A shipping style/behaviour law in the repo.** e.g. "two or three sentences, one breath
|
||||
each" → the ceiling belongs in the prompt as *words the model reads* ("at most three
|
||||
sentences; fewer if little happened"), not as `sqrt(chars / 1200)` in code.
|
||||
3. **A physical constant of the medium** (speech rate, packet size), measured against the
|
||||
actual device. Last resort: it is still a number you must defend.
|
||||
|
||||
Traps when parsing that config: commented decoy lines (`# contextWindow: 200,000` with a
|
||||
comma) must be skipped, so strip only lines whose trimmed form starts with `#`; the row
|
||||
scanner that returns on the `id:` line never sees the fields below it — keep scanning to the
|
||||
next `- id:`/provider key.
|
||||
|
||||
## 3. Encode the fix so the digit cannot come back
|
||||
|
||||
- Make the budget a **required parameter** of the pure module (no default: a module with no
|
||||
default cannot guess); the caller that resolved the endpoint does the derivation.
|
||||
- A derived budget that comes out non-positive is a **refusal** (`null`, zero requests),
|
||||
never a truncation — the caller's cursor/queue must hold and retry.
|
||||
- When an old cap must survive as a fallback for endpoints that declare nothing, rename it
|
||||
to what it is (`INHERITED_FLOOR`) and comment: floor, used only when nothing is declared.
|
||||
|
||||
## 4. Tests that pin the law, not the digit
|
||||
|
||||
- **Size-invariance:** assert the instruction/behaviour is *identical* for a tiny input and a
|
||||
huge one, proving volume no longer drives the removed ladder.
|
||||
- **Honours caller budget:** same span folded under an injected small window.
|
||||
- **Refusal, not truncation:** zero-window → null, zero model calls.
|
||||
- **DELETE** tests that pinned the deleted arithmetic; never re-pin them (house rule).
|
||||
- Pinning shipped prompt *words* is legitimate where the repo already does (prompt-contract
|
||||
test precedent); pinning an env/config value is a config assertion and belongs nowhere.
|
||||
|
||||
## 5. Report
|
||||
|
||||
Table first, verdict per number, and the net list of digits that remain after the change —
|
||||
each with its quoted source. If any digit survives with no source, that is the finding.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
name: media-journal-chain-verification
|
||||
description: "Verify a journal-guarded media relayout end to end: counts/bytes/size-mismatch from the JSONL journals, plus cross-journal accounting that proves \"missing\" destinations were moved onward rather than lost."
|
||||
---
|
||||
|
||||
# Verifying a media relayout across a chain of journals
|
||||
|
||||
`scripts/media-apply.py` (serverAppConfigs) writes a write-ahead JSONL journal beside the library root — one line per intent, `op` ∈ `mkdir|move`, a `move` record carrying `src`, `dst`, `bytes`, `mode`. Multiple runs over the same library produce **several journals** (relayout, then TV move, then duplicate staging), and the same file legitimately appears in more than one. Verifying one journal in isolation then reads as data loss.
|
||||
|
||||
## Verify counts and bytes from the journals, never from scratch files
|
||||
|
||||
`/tmp` and shell history are not records; the journals are. Read-only pass over every journal:
|
||||
|
||||
```python
|
||||
import json, os
|
||||
def recs(j): return [r for r in (json.loads(l) for l in open(j) if l.strip()) if r.get("op")=="move"]
|
||||
for j in journals:
|
||||
m = recs(j)
|
||||
gone = sum(1 for r in m if not os.path.exists(r["src"]))
|
||||
at = sum(1 for r in m if os.path.exists(r["dst"]))
|
||||
bad = sum(1 for r in m if os.path.exists(r["dst"])
|
||||
and os.path.getsize(r["dst"]) != r.get("bytes"))
|
||||
print(os.path.basename(j), len(m), sum(r.get("bytes",0) for r in m)/2**30, gone, at, bad)
|
||||
```
|
||||
|
||||
Assertions that matter: `src-gone == moves` (nothing half-moved), `size-mismatch == 0` (no truncation or clobber), `dst-present == moves` **for the newest journal only**.
|
||||
|
||||
## The chain step: account for every missing destination
|
||||
|
||||
An older journal's `dst` disappears when a later journal moves the file onward (episodes shelled out to the TV library, duplicates staged into `.AGENTTRASH`). Prove it instead of asserting it:
|
||||
|
||||
```python
|
||||
later = {}
|
||||
for j in later_journals:
|
||||
for r in recs(j): later.setdefault(r["src"], []).append(os.path.basename(j))
|
||||
missing = [r["dst"] for r in older_recs if not os.path.exists(r["dst"])]
|
||||
print(len(missing), sum(1 for d in missing if d in later),
|
||||
[d for d in missing if d not in later])
|
||||
```
|
||||
|
||||
Expect `unexplained = 0`. Measured 2026-10-30 on dankserver: 8,189 movie moves / 22,380 GiB with 174 destinations gone — exactly 124 (duplicate staging) + 6 (hand-filed strays) + 44 (mover episodes), `accounted-by-later-journal=174 unexplained=0`, zero size mismatches across all three journals.
|
||||
|
||||
## Post-purge consequence to state in the report
|
||||
|
||||
After an approved `.AGENTTRASH` purge the staging journal still parses, but `dst-present=0`: `media-apply.py rollback` on it recreates the folders **empty**. Say so — that journal is a directory-shape undo, not a byte restore; the keeper copy is the one that stayed in the library.
|
||||
|
||||
Do not trust a `dst-present` shortfall as loss until the cross-journal check has run, and never report counts from a `| grep -c` pipeline — mask-free exit status only.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: omp-session-journal-cursor-access
|
||||
description: "Read a live omp session's transcript mid-run from an extension (cursor = journal entry id via ctx.sessionManager), with the entry-type noise list, append lag, and the ctx.setInterval metronome."
|
||||
---
|
||||
|
||||
# Reading the live session journal from an extension
|
||||
|
||||
Use when an extension must summarise/narrate/index **everything said since some point**
|
||||
rather than reacting to one event payload, or when you need a durable "where I left off"
|
||||
pointer into a session. Verified 2026-10-30 with a throwaway probe under `omp -p`
|
||||
(v18.12.5), 1 s ticks over a two-tool-round run.
|
||||
|
||||
## The accessor (no plugin install needed)
|
||||
|
||||
`ctx.sessionManager` is on every `ExtensionContext` (`ReadonlySessionManager`,
|
||||
`@oh-my-pi/pi-coding-agent/dist/types/extensibility/extensions/types.d.ts:485`):
|
||||
|
||||
```
|
||||
getEntries() getBranch() getEntry(id) getLeafId() getLeafEntry()
|
||||
SessionMessageEntry = { id, type: "message", message: AgentMessage }
|
||||
```
|
||||
|
||||
- **The cursor is an entry id.** Persist `getLeafId()` (or the id of the last entry you
|
||||
actually consumed); "everything since" = entries after that id. Advance is O(1).
|
||||
- **`getEntries()` grows mid-run** — proven by a timer, with no handler help:
|
||||
`entries=3 → 6 → 7 → 9 → 11` while `idle=false`, leaf ids rotating.
|
||||
- **Append lags `message_end` by ~0.5–1 s.** So "the live edge" always under-reads by a
|
||||
second. Consume only up to the last id you actually read; the sub-second tail rides into
|
||||
the next window. Never advance past what you read.
|
||||
- **`ctx.setInterval(cb, ms)` / `ctx.setTimeout`** (`types.d.ts:555-572`): throws contained,
|
||||
`unref`'d, auto-cleared on `session_shutdown`. Use these, never raw `setInterval`.
|
||||
`ctx.isIdle()` is the cheap "is this session even working" guard.
|
||||
|
||||
## The journal is not the conversation (allowlist, never denylist)
|
||||
|
||||
`agent_end` reported `msgs=6` while the journal held `11` entries. Bookkeeping types in the
|
||||
union: `custom` (zero content, ~3 per tool round), `model_usage`, `thinking_level_change`,
|
||||
`mode_change`, `model_change`, `service_tier_change`, `label`, `title_change`,
|
||||
`ttsr_injection`, `session_init`, `credential_pin`, `reset_boundary`. Speech-relevant:
|
||||
`message` (roles `user` / `assistant` / `toolResult`) and, as context, `compaction` /
|
||||
`branch_summary`. New types appear on the harness's own schedule → an allowlist keeps them
|
||||
silent until someone decides they are speech.
|
||||
|
||||
Re-export the per-message readers instead of re-parsing content: an assistant message's
|
||||
narration-flagged `thinking` part is **answer text**, unflagged `thinking` is deliberation
|
||||
(`narration: true` discriminator).
|
||||
|
||||
## Robustness rules that came out of the failure modes
|
||||
|
||||
- Empty window ⇒ leave the cursor where it was (nothing new must not read as news).
|
||||
- Cursor id no longer in the journal (compaction, branch rewrite) ⇒ treat as "from the top
|
||||
of what is left", never as "nothing" — a lost pointer must not mute a session forever.
|
||||
`session_compact` and `session_branch` are dispatched events if you want to re-anchor eagerly.
|
||||
|
||||
## Probe recipe (proves any of the above in ~60 s, touches nothing live)
|
||||
|
||||
Project-local extensions load without installing: write `<run-dir>/.omp/extensions/probe.js`
|
||||
(default-export `function (pi)`), log to an absolute path, then
|
||||
|
||||
```bash
|
||||
: > /tmp/probe.log
|
||||
timeout 220 omp -p --thinking off --cwd <run-dir> \
|
||||
"Run these bash commands strictly one after the other: first: sleep 7; echo FIRST. then: sleep 7; echo SECOND. Then reply: FIRST SECOND"
|
||||
```
|
||||
|
||||
Register the tick inside `session_start` (`ctx.setInterval(() => stamp(info(ctx)), 1000)`),
|
||||
and give every entry you expect to exclude a speech-shaped payload with a sentinel word —
|
||||
an absence assertion with an empty producer proves nothing. Do NOT pass `--no-session`:
|
||||
the journal is the thing under test. Teardown: nothing to undo (no plugin link, no
|
||||
`~/.omp/plugins/package.json` write) — delete the run dir.
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: radiance-endpoint-silent-wedge-triage
|
||||
description: "Diagnose a radiance (frank-archon) endpoint that is wedged or crawling while systemd says active(running) and /health answers 200 — separate engine-state wedge from config, hardware, swap and clock causes, and prove the restart fixed it."
|
||||
---
|
||||
|
||||
# Silent wedge: `active (running)` + `/health` 200, but no throughput
|
||||
|
||||
For radiance endpoints (`frank-archon`, root unit whose `ExecStart` is `exec podman wait`). The unit can only exit when the container dies, so a stalled engine loop **never** triggers `Restart=always`, `up` stays `1`, and Alertmanager stays empty (every rule keys on `up==0`, `num_requests_waiting>0`, or TTFT p95). Verified 2026-10-30 21:00 on dankserver.
|
||||
|
||||
## Triage ladder (in this order — each step kills a class)
|
||||
|
||||
1. **Front end vs engine split.** `curl -w '%{time_total}' /health` vs `/v1/models`. A 0.4 ms `/health` beside a 20 s `/v1/models` timeout = front end alive, engine loop stalled.
|
||||
2. **Unit truth, not the port.** `systemctl status frank-archon -n 0` + `systemctl show -p MainPID,DropInPaths,ExecMainStartTimestamp`. A 4 h-old unit on a 2-week-old host means a restart happened today — cross-check `journalctl -u <svc> --since <day>` for `Starting/Stopping/exit` and `git log -- scripts/<svc>.*` to see whether a deploy caused it. Non-empty `DropInPaths` = leftover bench arm (separate trap).
|
||||
3. **Container storage namespace.** The unit is `User=dank`, so the container lives in **rootless** storage: `sudo podman ps -a` lists nothing and `podman inspect` says `no such object` — query as the user, without sudo.
|
||||
4. **Engine's own voice (the decisive artifact).** `podman logs --timestamps frank-archon | grep "E device:"`. `a synchronisation has seen no queue advance for 5 s` naming `barrier-and dep[0]=1` on both ranks = the TP all-reduce never completed. Timestamp the first occurrence; that is the incident start.
|
||||
5. **Rule out the engine's own suggested causes before believing anything else:** `cat /sys/module/amdgpu/parameters/cwsr_enable` (its message blames `cwsr_enable=0`; `1` falsifies it). Swap: read `/proc/<engine-pid>/status` VmSwap — do NOT infer from `free`/`/proc/swaps` totals (host-wide swap can be 32 GiB while the engine holds 378 MiB). Clocks during load: fire a request, then `rocm-smi --showclocks --showperflevel --showpower` + `/sys/class/drm/cardN/device/gpu_busy_percent` — 100 % busy at full sclk means neither throttle nor starvation.
|
||||
6. **Quantify the collapse from counters, not vibes.** `radiance:engine_step_seconds_decode_total` ÷ `radiance:engine_steps_decode_total` = mean decode step time; diff two scrapes (Δseconds ÷ Δsteps) for the current rate. Compare to the shape's recorded baseline in the endpoint's vault page (15.5 ms at frank-archon's 4×200000 fp8 shape; 209 ms = 13× collapse).
|
||||
|
||||
## Proof of fix (same-request A/B, not a health check)
|
||||
|
||||
Submit one fixed request before and after the restart and diff the engine's own `timings` object — an 8-token probe measures only first-token noise, use `max_tokens>=128`:
|
||||
|
||||
```
|
||||
curl -sS -m 240 -o /tmp/q.json -X POST :7770/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' -d @/tmp/query.json # then read timings.predicted_per_token_ms / predicted_per_second / ttft_ms / draft_n / draft_n_accepted
|
||||
```
|
||||
|
||||
Before/after on the identical request: 90.76 → 11.06 ms/token, 11.02 → 90.40 t/s, ttft 420 → 148 ms, acceptance unchanged (266/266). Then re-read the boot ledger (`elastic / kv cache / vram_weights / unclaimed / sampler rows`) and confirm it matches the page's recorded ledger — a successful restart proves nothing about the flags. Close with the restore oracle: `is-active` **and** `is-enabled` **and** no `/run/systemd/system/<unit>` mask symlink **and** container owned by the unit **and** `DropInPaths` empty.
|
||||
|
||||
## Law
|
||||
|
||||
A wedged radiance engine has **no in-place recovery** (its own wording: "a queue the driver evicted and could not restore") — `systemctl restart` is the only cure, and the fix must be proven by a rate, never by `/health`. Durable gap: no alert fires on a live-but-stalled endpoint; the missing rule is a decode-step-time ratio gated on `up==1` in `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Reference in new issue
Block a user