live: heartbeat sync 2026-10-29T15:29:37-07:00 (Refs #11)

agent/managed-skills/quant-pipeline-adversarial-review/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: quant-pipeline-adversarial-review
+description: "Review or audit the serverAppConfigs quantization campaign pipeline (convert-calib-kldref-score plus corpus builder/rechunker/pusher, fp8 stacker) for defects that silently waste GPU-hours or produce a number that is not the claimed number."
+---
+
+# Adversarial review of the quant campaign pipeline
+
+Friction-earned 2026-10-29 across `scripts/convert-qwen38-w5nl64.sh`,
+`run-qwen38-bf16.sh`, `make-calib-corpus.py`, `rechunk-kld-corpus.py`,
+`calib-push.py`, `stack-qwen38-official-fp8.py`. Every class below was a real
+defect in that tree. Review code, not vibes: quote the line, name the cost.
+
+## Files in scope (one campaign = this set)
+
+| Stage | File | Owns |
+|---|---|---|
+| corpus | `make-calib-corpus.py` | calib + kld JSONL, manifest |
+| corpus | `rechunk-kld-corpus.py` | windows that fit the container's KV window |
+| convert | `convert-<model>.sh` | plan → artifact per arm |
+| tape | `run-qwen38-bf16.sh` | the rig's argv, mount, trap, gates |
+| drive | `calib-push.py` | prefill traffic into the tap |
+| fp8 path | `stack-qwen38-official-fp8.py` | expert stacking + PLE merge |
+
+## Defect classes to hunt, in cost order
+
+1. **Claimed identity that is only a length check.** A tool that writes TEXT back
+   out (`tok.decode(window)`) and asserts only counts is not proving token
+   identity — decode→encode can merge/split. Fix: re-encode each window and
+   compare id lists; refuse on mismatch. Same shape: an assertion on "N tokens
+   in, N tokens out" that counts windows rather than re-encoded text.
+2. **A loop that cannot finish and cannot notice.** Drivers that re-push a corpus
+   `--max-rounds` times without measuring the sink. Compare the tap/output dir's
+   byte total per round and abort on zero growth; abort on zero successes; abort
+   on zero prompts. Without this the loss is silent for hours.
+3. **`set -u` argument holes.** `--flag) VAR="$2"; shift 2` dies with "unbound
+   variable" when the flag is last. Guard `[ "$#" -ge 2 ]` in every such arm.
+4. **Restore paths that leave the GPU holder alive.** `--rm` only fires when
+   podman itself exits, so a killed run leaves conmon holding the cards while
+   `systemctl is-active` says `active`. `restore()` MUST `podman rm -f "$C"`
+   before starting production. Prove with `podman ps` after a `kill`.
+5. **Gates that run after the expensive phase.** A leftover container name, an
+   existing output, a non-writable tap dir, an unset figure — check them before
+   the plan/boot, not when podman/sed fails mid-run.
+6. **Zero-exit with no artifact.** After `rc=0`, assert the output file exists
+   before `stat`/arithmetic on it.
+7. **`assert` in a verifier.** `python -O` deletes it silently. Use `raise`.
+8. **Warning where the code's own assumption breaks.** e.g. a 128-lane pack fed
+   an HD that is not a multiple of 128 — warning + continue writes a wrong table.
+9. **Re-tokenization in loops.** A pool cycled 40× re-tokenizing each fragment
+   per pass; and a manifest recomputing a total by re-encoding the whole corpus.
+   Cache counts once, accumulate during generation.
+10. **Substring tests standing in for key tests.** `"content" in line` before
+    `json.loads(line)["content"]` crashes on any list-shaped `.json`.
+11. **Path guards compared as strings.** `A/../A/x` defeats an equality check;
+    add `realpath -m` on both sides.
+12. **Shellcheck's honest false positive**: single-quoted `$CALIB` passed to a
+    converter that expands it from the container env is correct (SC2016). Say so
+    in the PR; do not "fix" it.
+
+## Procedure
+
+1. Inventory: `wc -l` the scripts + recipes; read each **completely** (ranges,
+   not whole files, but every line). Recipes carry the numbers scripts assert.
+2. Read the tests that pin the argv (`tests/qwen38_reference.bats`'s `SHAPE`
+   fixture, `tests/qwen38_campaign.bats`) — they are the contract you must not
+   break, and the first place a stale number hides.
+3. Mechanical gates:
+   ```bash
+   python3 -m py_compile scripts/*.py
+   bash -n scripts/<each>.sh
+   shellcheck -S warning scripts/<each>.sh      # never pass .py to it
+   ./tests/run.sh
+   ```
+4. Fix the real ones only; keep diffs surgical; re-run the suite after the batch.
+5. Smoke the changed path for real — smoke beats the suite:
+   ```bash
+   # run a local script on the server's data via the corpus venv, no checkout needed
+   ssh dankserver '/home/dank/venvs/radcorpus/bin/python - --corpus <corpus> --out /tmp/<x> --force' \
+     < scripts/rechunk-kld-corpus.py
+   ```
+   Import-level helpers can be exercised with
+   `importlib.util.spec_from_file_location` (e.g. a new `tap_bytes()`: empty→0,
+   one file→its size, absent→0).
+6. Stacked PRs: a review commit goes on its own branch off the open one; a diff
+   that includes the parent commit is a stacked PR, not a mistake — say which PR
+   merges first in the body.
+
+## Report
+
+One table of defect → cost → fix. Name the failure the defect produced in
+practice (hours wasted, wrong number), because that is what makes it reviewable.