From ad5903c2945004031744b48808fa5834fbba5570 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 30 Oct 2026 14:17:40 -0700 Subject: [PATCH] live: heartbeat sync 2026-10-30T14:17:40-07:00 (Refs #11) --- .../agenttrash-purger-invocation/SKILL.md | 25 ++++++++---- .../radiance-w4-campaign/SKILL.md | 38 +++++++++---------- .../state.json | 2 +- .../.local/state/noctalia/settings.toml | 10 ++--- 4 files changed, 41 insertions(+), 34 deletions(-) diff --git a/agent/managed-skills/agenttrash-purger-invocation/SKILL.md b/agent/managed-skills/agenttrash-purger-invocation/SKILL.md index 73ec5a5..6d5c90b 100644 --- a/agent/managed-skills/agenttrash-purger-invocation/SKILL.md +++ b/agent/managed-skills/agenttrash-purger-invocation/SKILL.md @@ -1,9 +1,9 @@ --- name: agenttrash-purger-invocation -description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) to clear staged deletions, and diagnose its guaranteed exit 1 on hosts whose find is uutils (Rust) rather than GNU." +description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) to clear staged deletions, and diagnose its two refusal traps: a false stale-lock error on a full filesystem, and $HOME refused on dankserver by podman-storage permission denials." --- -# Purging .AGENTTRASH — invocation and the uutils-find trap +# Purging .AGENTTRASH — invocation and two refusal traps `scripts/purge-agenttrash.sh` (serverAppConfigs, merged in PR #79 `fad87aa`) is the tool form of the fleet's deletion policy: files land in `.AGENTTRASH`, and an approved purge runs this instead of being improvised per session. @@ -19,10 +19,21 @@ sh scripts/purge-agenttrash.sh --execute /some/root # scope to one root (defaul - Every printable line is fixed at plan time in `$HOME/.agenttrash-purge.lock/plan`, so dry-run and `--execute` output cannot drift. Exit `0` ok · `1` any refusal or failure · `2` usage. - Rule: a root's own `.AGENTTRASH` is *emptied* (the directory survives); deeper `.AGENTTRASH` trees are *removed*; symlinked or foreign-owned candidates are refused, never followed; `/` is refused outright. -## The trap: uutils find makes it exit 1 on the dev box +## Trap 1: a FULL filesystem makes it lie about a lock -On the workstation, `sh scripts/purge-agenttrash.sh` reports `refused /home/blank (find failed)` and exits 1 **even though the plan is correct** — and an `--execute` there acts on the recorded plan while still reporting failure. +The script acquires `~/.agenttrash-purge.lock` and writes a plan file **before** it scans. On a filesystem with 0 bytes free both writes fail `ENOSPC`, and the run dies with: -- Cause: `find --version` reports `find 0.8.0` (Rust **uutils**, errors as `Error: : Permission denied (os error 13)`), not GNU findutils. It exits 1 whenever any directory under the root is unreadable — and `~/.local/share/containers/storage/**` (podman overlay layers, mode 0700 dirs) is unreadable. `collect_root` turns any nonzero find status into a root refusal. -- Proved by: `find /home/blank -xdev -name .AGENTTRASH >/dev/null 2>&1; echo $?` → `1`, while `find /home/blank/.omp -xdev ...` → `0`. -- Fix, if it ever comes up (don't invent it): treat unreadable-directory errors as warnings, or accept find's partial output and reserve refusal for a hard find failure. Note `bash -n` is not the gate — shellcheck / `sh -n` is. +``` +purge-agenttrash: /home/…/.agenttrash-purge.lock exists and is not ours to reclaim +``` + +while **no lock exists** (`find -maxdepth 2 -name '.agenttrash-purge.lock'` comes back empty, `pgrep -af purge-agenttrash` shows nothing). Do not hunt the phantom lock and do not delete anything to "reclaim" it: check `df -h` first. Free space by the purge's own intended action — delete the staged item the operator approved (a same-filesystem `mv` into `.AGENTTRASH` frees nothing; only deletion does) — then re-run the tool, which will clean the rest and log its summary. + +## Trap 2: `$HOME` is always refused on dankserver + +`sh scripts/purge-agenttrash.sh` on dankserver reports `refused /home/dank (find failed)` and exits 1 even though the plan is otherwise right, and `~/.AGENTTRASH-cleanup.log` records `refused=1` beside whatever it did manage. + +- Cause: `find /home/dank -xdev -name .AGENTTRASH` exits 1 there because rootless podman storage contains UID-mismatched directories — `~/.local/share/containers/storage/overlay/*/diff/var/lib/mysql`, `…/etc/valkey`, `…/.config/calibre/plugins`, `…/var/cache/apt/archives/partial` — plus `~/Containers/romm/mariadb/*`. `collect_root` turns any nonzero find status into a root refusal, so a *partial* scan becomes no plan. +- Find still prints every real tree (14 on dankserver: `vLLM/`, `Containers/`, four under `serverAppConfigs`, the swift15 checkout), so the refusal is about the scan status, not the candidates. +- This is the same class as the uutils-find trap on the dev box, with a different cause (GNU find + foreign-owned overlay internals). Minimal fix, safe because a `.AGENTTRASH` never lives inside container storage: prune `-path "$HOME/.local/share/containers"` from the scan, or accept find's partial output and reserve the refusal for a hard find failure. Verify with `sh -n`/shellcheck, not `bash -n`. +- Do not run the purger with `sudo` to dodge this: candidates are judged by ownership, and a root-owned scan changes the refusal outcome rather than fixing it. Retire the blocker, then re-run as the deploy user. diff --git a/agent/managed-skills/radiance-w4-campaign/SKILL.md b/agent/managed-skills/radiance-w4-campaign/SKILL.md index 57583dd..93011b4 100644 --- a/agent/managed-skills/radiance-w4-campaign/SKILL.md +++ b/agent/managed-skills/radiance-w4-campaign/SKILL.md @@ -1,31 +1,27 @@ --- name: radiance-w4-campaign -description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws." +description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin, tap-engagement and MNBT laws." --- -Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/Quantization Production.md` + `scripts/` (`calib-push.py`, `qwen38-calib-window.sh`, `run-qwen38-bf16.sh`, `derive-protected-experts.py`, `apply-protected-cast.sh`, the convert script, the KLD gate, the fleet probes). +Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`, `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`). Sequence: 1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv). -2. **BF16 container**: `rad-convert` no-recipe (pure copy) — the KLD reference engine and the calibration rig. -3. **KLD reference**: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (`rows:` line = receipt). -4. **Calibration**: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as `RADGRAM2` files (`gram.r...bin`). -5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's `# PROTECTED-BEGIN/END` markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split. -6. **Final convert** on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first). -7. **KLD gate**: MNBT ≤ 2048 when a w4 container streams in place at 32768 (`DID NOT FIT … stored form is not the container's`), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %. -8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + `reasoning_content` fallback). +2. **BF16 container**: `rad-convert` no-recipe (pure copy) — serves as the KLD reference engine. +3. **KLD reference**: record logits on the SERVE pin (1.0.13) at MNBT 32768, `--max-model-len 16384`, corpus ≤16k prompts. Writes atomically at completion (`rows:` line = receipt). Never scp a running bash script (lazy-read law). +4. **Calibration**: the tap arms on the **1.2.0** pin — see the section below before any hour is spent. +5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between `# PROTECTED-BEGIN/END` markers of the recipe. Plugin law: ONE encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split. +6. **Final convert** on the SERVE pin (CPU GPTQ, hours; disk gate ≥146 GiB free first). +7. **KLD gate** on the serve pin: **MNBT ≤ 2048** (w4 streams in place only there; 32768 → `DID NOT FIT ... stored form is not the container's`), pool 40960, fp8 KV matching the production KV dtype. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %. +8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id = served-model-name (alias independent of file — harness models.yml untouched), boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥256-token budget + `reasoning_content` fallback). -## The calibration pin law, and how a tap dies silently (measured 2026-10-30) +## The calibration tap: pin, arming, engagement gate, output size -- **`calib` MUST boot the 1.2.0 pin (digest `sha256:665775ed…`), never the production pin.** Setting `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` on 1.8.1 looks accepted and writes **nothing**: 17 h and 7.3 M prompt tokens produced zero bytes, with `/health` green, 118 tok/s and no error line. The tap code (`CalibExperts`, `RADGRAM2`, the seal line `calibration wrote layer Hessians after %lld tokens`) is present in every plugin from 1.2.4 through 1.8.1 — **string presence is not the arm**. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins `IMAGE` per mode (`calib` → `PIN_120`) and the qwen38 runner's single `FRANK_ARCHON_IMAGE` port was a silent regression. -- **The boot ledger's `ENVIRONMENT -- the RADIANCE_* variables set, by name` block is a dump, not acceptance.** It lists names the engine may ignore. -- **`/stats` has no calibration counter** and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an **engagement gate**: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory. -- Calibration at `--tp 2` is legal (`expert_tiered` + `--weights-disk-tier` streams whole experts); the plugin refuses only when a rank would hold a *slice* of every expert ("calibration runs at one rank … its Hessians would be of the slice"). -- The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved **118 tok/s vs ~94, ≈1.2×**, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (`total_requests` deltas × corpus mean tokens/doc); **never** from `prefill_tps`, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure). +- **Pin: 1.2.0 arms the tap, the production pin does not.** `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` are the whole interface (there is no `--calib*` flag on any pin). Running a tap on the 1.8.1 serve image produced **zero bytes over 17 h / 7.3 M prompt tokens** at a healthy 118 tok/s — no error, no ledger line, no `/stats` counter. Swift15's runner pins `PIN_120` for the `calib` mode only; a runner that uses `FRANK_ARCHON_IMAGE` for every mode silently calibrates into a void. +- **Strings prove code, not arming.** `CalibExperts`, `RADGRAM2` and the seal line `calibration wrote layer Hessians after %lld tokens` exist in `architectures/qwen4exp_fp8.so` from 1.2.4 through 1.8.1, and the boot ledger lists RADIANCE_* env **names** (a dump, not support). Neither is evidence the tap is engaged. +- **Engagement gate, before any long drain.** Boot with a tiny floor (`RADIANCE_CALIB_TOKENS=8192`), push 2–3 corpus documents, and require the seal line plus bytes in the tap dir. The seal fires after ~10 k tokens regardless of prompt size, so the test costs minutes — the absence of this check cost 17 h. +- **One seal writes ~40 GB** (per-expert gram + Hessian set, sized by expert width and layer count, essentially independent of how few tokens triggered it). Point `RADIANCE_CALIB_DIR` at a volume with ≥60 GiB free (`/mnt/entropy` on dankserver) and gate on free space before boot; landing it on a 37 GiB root filled the filesystem, and every subsequent write came back `was short` / `cannot write`. +- The tap needs the bf16 container (`calibrate the bf16 container`), tp=2 with `--placement expert_tiered --weights-disk-tier` is fine (swift15 did it), and a sliced-expert layout is refused by name (`calibration runs at one rank; this rank holds a slice of every expert's width`). 1.2.0 lacks `--flash-attn`, `--metrics`, `--expert-nl`, `--busy-prefill-chunk`; it does have every measuring-shape flag (`--kv-cache-dtype`, `--max-model-len`, `--ngram-placement`, `--num-speculative-tokens`, `--override-chat-template`). +- Engine rates: `prefill_tps` in `/stats` is a **rolling gauge**, never a cumulative counter — derive throughput from `total_requests × corpus mean tokens/doc`, or from `steps_prefill`. A wide calib step (MNBT 8192) turns ~2.5 steps/doc into 1.0 and buys ~1.2× on a disk-tiered bf16 rig, not 4.5×. -## Ops laws that bite - -- **`systemctl` on the system units needs `sudo`** in scripts and over ssh. Bare `systemctl stop frank-archon` passes one day and returns `Access denied … requires interactive authentication` the next, and a runner that ignores the failure boots the rig *beside* live production. Check `systemctl is-active` after every stop/start and fail closed. -- Stopping the unit leaves it `failed` (the engine exits non-zero on SIGTERM); `systemctl reset-failed` after the stop keeps a deliberate drain from reading as a fault. -- A probe container started with `--rm` deletes its own log on exit; capture `podman logs` before cleanup, and never redirect `podman run -d` to a file expecting the engine's output — that file only receives the container id. -- Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against `podman inspect --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them down. +Laws: stop frank-archon for every GPU phase (relinked unit: stop, mask fails; plain `systemctl` for a system unit is denied by polkit — use `sudo systemctl`); verify docs against `podman inspect --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing shapes; ff-pull fails on untracked files the merge tracks — trash first. diff --git a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json index 8a8b127..23db0db 100644 --- a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json +++ b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1793391054728, + "lastConsolidatedAt": 1793395014767, "lastResult": { "at": 1793315963987, "sessions": 1, diff --git a/dotfiles/noctalia/.local/state/noctalia/settings.toml b/dotfiles/noctalia/.local/state/noctalia/settings.toml index ca80a54..ed28f59 100644 --- a/dotfiles/noctalia/.local/state/noctalia/settings.toml +++ b/dotfiles/noctalia/.local/state/noctalia/settings.toml @@ -279,22 +279,22 @@ transition = [ "fade" ] interval_seconds = 1200 [wallpaper.default] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" [wallpaper.last] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" [wallpaper.monitors.DP-1] path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png" [wallpaper.monitors.DP-2] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" [wallpaper.monitors.HDMI-A-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" [wallpaper.monitors.eDP-1] - path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png" + path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png" [weather] unit = "imperial"