live: heartbeat sync 2026-10-30T14:17:40-07:00 (Refs #11)
This commit is contained in:
Frank committed
1 parent
ef7478f9c7
commit
ad5903c294
4 files changed
+41
-34
No files matched your search
@@ -1,9 +1,9 @@
|
||||
---
|
||||
name: agenttrash-purger-invocation
|
||||
description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) to clear staged deletions, and diagnose its guaranteed exit 1 on hosts whose find is uutils (Rust) rather than GNU."
|
||||
description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) to clear staged deletions, and diagnose its two refusal traps: a false stale-lock error on a full filesystem, and $HOME refused on dankserver by podman-storage permission denials."
|
||||
---
|
||||
|
||||
# Purging .AGENTTRASH — invocation and the uutils-find trap
|
||||
# Purging .AGENTTRASH — invocation and two refusal traps
|
||||
|
||||
`scripts/purge-agenttrash.sh` (serverAppConfigs, merged in PR #79 `fad87aa`) is the tool form of the fleet's deletion policy: files land in `.AGENTTRASH`, and an approved purge runs this instead of being improvised per session.
|
||||
|
||||
@@ -19,10 +19,21 @@ sh scripts/purge-agenttrash.sh --execute /some/root # scope to one root (defaul
|
||||
- Every printable line is fixed at plan time in `$HOME/.agenttrash-purge.lock/plan`, so dry-run and `--execute` output cannot drift. Exit `0` ok · `1` any refusal or failure · `2` usage.
|
||||
- Rule: a root's own `.AGENTTRASH` is *emptied* (the directory survives); deeper `.AGENTTRASH` trees are *removed*; symlinked or foreign-owned candidates are refused, never followed; `/` is refused outright.
|
||||
|
||||
## The trap: uutils find makes it exit 1 on the dev box
|
||||
## Trap 1: a FULL filesystem makes it lie about a lock
|
||||
|
||||
On the workstation, `sh scripts/purge-agenttrash.sh` reports `refused /home/blank (find failed)` and exits 1 **even though the plan is correct** — and an `--execute` there acts on the recorded plan while still reporting failure.
|
||||
The script acquires `~/.agenttrash-purge.lock` and writes a plan file **before** it scans. On a filesystem with 0 bytes free both writes fail `ENOSPC`, and the run dies with:
|
||||
|
||||
- Cause: `find --version` reports `find 0.8.0` (Rust **uutils**, errors as `Error: <path>: Permission denied (os error 13)`), not GNU findutils. It exits 1 whenever any directory under the root is unreadable — and `~/.local/share/containers/storage/**` (podman overlay layers, mode 0700 dirs) is unreadable. `collect_root` turns any nonzero find status into a root refusal.
|
||||
- Proved by: `find /home/blank -xdev -name .AGENTTRASH >/dev/null 2>&1; echo $?` → `1`, while `find /home/blank/.omp -xdev ...` → `0`.
|
||||
- Fix, if it ever comes up (don't invent it): treat unreadable-directory errors as warnings, or accept find's partial output and reserve refusal for a hard find failure. Note `bash -n` is not the gate — shellcheck / `sh -n` is.
|
||||
```
|
||||
purge-agenttrash: /home/…/.agenttrash-purge.lock exists and is not ours to reclaim
|
||||
```
|
||||
|
||||
while **no lock exists** (`find <root> -maxdepth 2 -name '.agenttrash-purge.lock'` comes back empty, `pgrep -af purge-agenttrash` shows nothing). Do not hunt the phantom lock and do not delete anything to "reclaim" it: check `df -h` first. Free space by the purge's own intended action — delete the staged item the operator approved (a same-filesystem `mv` into `.AGENTTRASH` frees nothing; only deletion does) — then re-run the tool, which will clean the rest and log its summary.
|
||||
|
||||
## Trap 2: `$HOME` is always refused on dankserver
|
||||
|
||||
`sh scripts/purge-agenttrash.sh` on dankserver reports `refused /home/dank (find failed)` and exits 1 even though the plan is otherwise right, and `~/.AGENTTRASH-cleanup.log` records `refused=1` beside whatever it did manage.
|
||||
|
||||
- Cause: `find /home/dank -xdev -name .AGENTTRASH` exits 1 there because rootless podman storage contains UID-mismatched directories — `~/.local/share/containers/storage/overlay/*/diff/var/lib/mysql`, `…/etc/valkey`, `…/.config/calibre/plugins`, `…/var/cache/apt/archives/partial` — plus `~/Containers/romm/mariadb/*`. `collect_root` turns any nonzero find status into a root refusal, so a *partial* scan becomes no plan.
|
||||
- Find still prints every real tree (14 on dankserver: `vLLM/`, `Containers/`, four under `serverAppConfigs`, the swift15 checkout), so the refusal is about the scan status, not the candidates.
|
||||
- This is the same class as the uutils-find trap on the dev box, with a different cause (GNU find + foreign-owned overlay internals). Minimal fix, safe because a `.AGENTTRASH` never lives inside container storage: prune `-path "$HOME/.local/share/containers"` from the scan, or accept find's partial output and reserve the refusal for a hard find failure. Verify with `sh -n`/shellcheck, not `bash -n`.
|
||||
- Do not run the purger with `sudo` to dodge this: candidates are judged by ownership, and a root-owned scan changes the refusal outcome rather than fixing it. Retire the blocker, then re-run as the deploy user.
|
||||
@@ -1,31 +1,27 @@
|
||||
---
|
||||
name: radiance-w4-campaign
|
||||
description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin and MNBT laws."
|
||||
description: "Run a radiance GPTQ-w4 quantization campaign on dankserver: calibration → protected experts → convert → KLD gate → deploy, with the engine-pin, tap-engagement and MNBT laws."
|
||||
---
|
||||
|
||||
Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/Quantization Production.md` + `scripts/` (`calib-push.py`, `qwen38-calib-window.sh`, `run-qwen38-bf16.sh`, `derive-protected-experts.py`, `apply-protected-cast.sh`, the convert script, the KLD gate, the fleet probes).
|
||||
Pipeline for quantizing a BF16 Qwen3.8-Flash-Next checkpoint into a servable `qwen4exp` `.rad` on dankserver (frank-primary). Reference run: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`; scripts in `scripts/` (`run-swift15-bf16.sh`, `calib-push.py`, `derive-protected-experts.py`, `apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`, `kld-gate-swift15.sh`, `fleet-probes-swift15.sh`).
|
||||
|
||||
Sequence:
|
||||
1. **Fetch** checkpoint with per-shard LFS sha256 verification (`-f`, byte counts, no token on argv).
|
||||
2. **BF16 container**: `rad-convert` no-recipe (pure copy) — the KLD reference engine and the calibration rig.
|
||||
3. **KLD reference**: record logits on the serving shape, corpus ≤ the per-slot window. Writes atomically at completion (`rows:` line = receipt).
|
||||
4. **Calibration**: see the pin law below. Push the corpus through prefill; per-expert down-proj Grams land as `RADGRAM2` files (`gram.r<rank>.<tensor>.<e>.bin`).
|
||||
5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between the recipe's `# PROTECTED-BEGIN/END` markers. One encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
|
||||
6. **Final convert** on the serving pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
|
||||
7. **KLD gate**: MNBT ≤ 2048 when a w4 container streams in place at 32768 (`DID NOT FIT … stored form is not the container's`), pool 40960, KV dtype matching production. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
|
||||
8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id, boot ledger, a streamed ≥128-token decode probe, and fleet probes (vision needs ≥512 budget on a thinking model + `reasoning_content` fallback).
|
||||
2. **BF16 container**: `rad-convert` no-recipe (pure copy) — serves as the KLD reference engine.
|
||||
3. **KLD reference**: record logits on the SERVE pin (1.0.13) at MNBT 32768, `--max-model-len 16384`, corpus ≤16k prompts. Writes atomically at completion (`rows:` line = receipt). Never scp a running bash script (lazy-read law).
|
||||
4. **Calibration**: the tap arms on the **1.2.0** pin — see the section below before any hour is spent.
|
||||
5. **Protected experts**: derive from gram energy (share ≥3 % AND ≥3× routing share), splice between `# PROTECTED-BEGIN/END` markers of the recipe. Plugin law: ONE encoding per expert table; only whole bf16 experts legal beside it — no gate_up/down split.
|
||||
6. **Final convert** on the SERVE pin (CPU GPTQ, hours; disk gate ≥146 GiB free first).
|
||||
7. **KLD gate** on the serve pin: **MNBT ≤ 2048** (w4 streams in place only there; 32768 → `DID NOT FIT ... stored form is not the container's`), pool 40960, fp8 KV matching the production KV dtype. Gate: mean KL ≤ 0.03, top-1 ≥ 95 %.
|
||||
8. **Deploy** `./scripts/deploy.sh dankserver --service frank-archon`; verify `/v1/models` id = served-model-name (alias independent of file — harness models.yml untouched), boot ledger, streamed ≥128-token decode probe, fleet probes (vision needs ≥256-token budget + `reasoning_content` fallback).
|
||||
|
||||
## The calibration pin law, and how a tap dies silently (measured 2026-10-30)
|
||||
## The calibration tap: pin, arming, engagement gate, output size
|
||||
|
||||
- **`calib` MUST boot the 1.2.0 pin (digest `sha256:665775ed…`), never the production pin.** Setting `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` on 1.8.1 looks accepted and writes **nothing**: 17 h and 7.3 M prompt tokens produced zero bytes, with `/health` green, 118 tok/s and no error line. The tap code (`CalibExperts`, `RADGRAM2`, the seal line `calibration wrote layer Hessians after %lld tokens`) is present in every plugin from 1.2.4 through 1.8.1 — **string presence is not the arm**. Only the 1.2.0 plugin arms on the env pair, which is why swift15's runner pins `IMAGE` per mode (`calib` → `PIN_120`) and the qwen38 runner's single `FRANK_ARCHON_IMAGE` port was a silent regression.
|
||||
- **The boot ledger's `ENVIRONMENT -- the RADIANCE_* variables set, by name` block is a dump, not acceptance.** It lists names the engine may ignore.
|
||||
- **`/stats` has no calibration counter** and the tap dir stays empty until the floor is reached, so a dead tap is indistinguishable from a slow one for hours. Therefore every tap run needs an **engagement gate**: push 2–3 documents, then refuse to continue unless the tap dir holds bytes. Put it in the runner, not in an operator's memory.
|
||||
- Calibration at `--tp 2` is legal (`expert_tiered` + `--weights-disk-tier` streams whole experts); the plugin refuses only when a rank would hold a *slice* of every expert ("calibration runs at one rank … its Hessians would be of the slice").
|
||||
- The reported gain from a wide calib step is small: MNBT 2,048 → 8,192 cut steps/doc 2.5 → 1.0 and moved **118 tok/s vs ~94, ≈1.2×**, because the tap's cost is bytes read from the disk tier, not steps. Size the token floor from a counter measurement (`total_requests` deltas × corpus mean tokens/doc); **never** from `prefill_tps`, which is a rolling gauge (it manufactured a false 4.5× / 320 tok/s figure).
|
||||
- **Pin: 1.2.0 arms the tap, the production pin does not.** `RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS` are the whole interface (there is no `--calib*` flag on any pin). Running a tap on the 1.8.1 serve image produced **zero bytes over 17 h / 7.3 M prompt tokens** at a healthy 118 tok/s — no error, no ledger line, no `/stats` counter. Swift15's runner pins `PIN_120` for the `calib` mode only; a runner that uses `FRANK_ARCHON_IMAGE` for every mode silently calibrates into a void.
|
||||
- **Strings prove code, not arming.** `CalibExperts`, `RADGRAM2` and the seal line `calibration wrote layer Hessians after %lld tokens` exist in `architectures/qwen4exp_fp8.so` from 1.2.4 through 1.8.1, and the boot ledger lists RADIANCE_* env **names** (a dump, not support). Neither is evidence the tap is engaged.
|
||||
- **Engagement gate, before any long drain.** Boot with a tiny floor (`RADIANCE_CALIB_TOKENS=8192`), push 2–3 corpus documents, and require the seal line plus bytes in the tap dir. The seal fires after ~10 k tokens regardless of prompt size, so the test costs minutes — the absence of this check cost 17 h.
|
||||
- **One seal writes ~40 GB** (per-expert gram + Hessian set, sized by expert width and layer count, essentially independent of how few tokens triggered it). Point `RADIANCE_CALIB_DIR` at a volume with ≥60 GiB free (`/mnt/entropy` on dankserver) and gate on free space before boot; landing it on a 37 GiB root filled the filesystem, and every subsequent write came back `was short` / `cannot write`.
|
||||
- The tap needs the bf16 container (`calibrate the bf16 container`), tp=2 with `--placement expert_tiered --weights-disk-tier` is fine (swift15 did it), and a sliced-expert layout is refused by name (`calibration runs at one rank; this rank holds a slice of every expert's width`). 1.2.0 lacks `--flash-attn`, `--metrics`, `--expert-nl`, `--busy-prefill-chunk`; it does have every measuring-shape flag (`--kv-cache-dtype`, `--max-model-len`, `--ngram-placement`, `--num-speculative-tokens`, `--override-chat-template`).
|
||||
- Engine rates: `prefill_tps` in `/stats` is a **rolling gauge**, never a cumulative counter — derive throughput from `total_requests × corpus mean tokens/doc`, or from `steps_prefill`. A wide calib step (MNBT 8192) turns ~2.5 steps/doc into 1.0 and buys ~1.2× on a disk-tiered bf16 rig, not 4.5×.
|
||||
|
||||
## Ops laws that bite
|
||||
|
||||
- **`systemctl` on the system units needs `sudo`** in scripts and over ssh. Bare `systemctl stop frank-archon` passes one day and returns `Access denied … requires interactive authentication` the next, and a runner that ignores the failure boots the rig *beside* live production. Check `systemctl is-active` after every stop/start and fail closed.
|
||||
- Stopping the unit leaves it `failed` (the engine exits non-zero on SIGTERM); `systemctl reset-failed` after the stop keeps a deliberate drain from reading as a fault.
|
||||
- A probe container started with `--rm` deletes its own log on exit; capture `podman logs` before cleanup, and never redirect `podman run -d` to a file expecting the engine's output — that file only receives the container id.
|
||||
- Stop the production service for every GPU phase (relinked unit: stop, mask fails); verify shapes against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing them down.
|
||||
Laws: stop frank-archon for every GPU phase (relinked unit: stop, mask fails; plain `systemctl` for a system unit is denied by polkit — use `sudo systemctl`); verify docs against `podman inspect <name> --format {{join .Config.Cmd " "}}` (rootless, as dank) before writing shapes; ff-pull fails on untracked files the merge tracks — trash first.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"v": 1,
|
||||
"lastConsolidatedAt": 1793391054728,
|
||||
"lastConsolidatedAt": 1793395014767,
|
||||
"lastResult": {
|
||||
"at": 1793315963987,
|
||||
"sessions": 1,
|
||||
|
||||
@@ -279,22 +279,22 @@ transition = [ "fade" ]
|
||||
interval_seconds = 1200
|
||||
|
||||
[wallpaper.default]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
|
||||
|
||||
[wallpaper.last]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
|
||||
|
||||
[wallpaper.monitors.DP-1]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
|
||||
|
||||
[wallpaper.monitors.DP-2]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
|
||||
|
||||
[wallpaper.monitors.HDMI-A-1]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
|
||||
|
||||
[wallpaper.monitors.eDP-1]
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
|
||||
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gw26v3.png"
|
||||
|
||||
[weather]
|
||||
unit = "imperial"
|
||||
|
||||
Reference in new issue
Block a user