command-center/agent/managed-skills/quant-pipeline-adversarial-review/SKILL.md

name, description

name	description
quant-pipeline-adversarial-review	Review or audit the serverAppConfigs quantization campaign pipeline (convert-calib-kldref-score plus corpus builder/rechunker/pusher, fp8 stacker) for defects that silently waste GPU-hours or produce a number that is not the claimed number.

Adversarial review of the quant campaign pipeline

Friction-earned 2026-10-29 across scripts/convert-qwen38-w5nl64.sh, run-qwen38-bf16.sh, make-calib-corpus.py, rechunk-kld-corpus.py, calib-push.py, stack-qwen38-official-fp8.py. Every class below was a real defect in that tree. Review code, not vibes: quote the line, name the cost.

Files in scope (one campaign = this set)

Stage	File	Owns
corpus	make-calib-corpus.py	calib + kld JSONL, manifest
corpus	rechunk-kld-corpus.py	windows that fit the container's KV window
convert	convert-<model>.sh	plan → artifact per arm
tape	run-qwen38-bf16.sh	the rig's argv, mount, trap, gates
drive	calib-push.py	prefill traffic into the tap
fp8 path	stack-qwen38-official-fp8.py	expert stacking + PLE merge

Defect classes to hunt, in cost order

1. Claimed identity that is only a length check. A tool that writes TEXT back out (tok.decode(window)) and asserts only counts is not proving token identity — decode→encode can merge/split. Fix: re-encode each window and compare id lists; refuse on mismatch. Same shape: an assertion on "N tokens in, N tokens out" that counts windows rather than re-encoded text.
2. A loop that cannot finish and cannot notice. Drivers that re-push a corpus --max-rounds times without measuring the sink. Compare the tap/output dir's byte total per round and abort on zero growth; abort on zero successes; abort on zero prompts. Without this the loss is silent for hours.
3. set -u argument holes. --flag) VAR="$2"; shift 2 dies with "unbound variable" when the flag is last. Guard [ "$#" -ge 2 ] in every such arm.
4. Restore paths that leave the GPU holder alive. --rm only fires when podman itself exits, so a killed run leaves conmon holding the cards while systemctl is-active says active. restore() MUST podman rm -f "$C" before starting production. Prove with podman ps after a kill.
5. Gates that run after the expensive phase. A leftover container name, an existing output, a non-writable tap dir, an unset figure — check them before the plan/boot, not when podman/sed fails mid-run.
6. Zero-exit with no artifact. After rc=0, assert the output file exists before stat/arithmetic on it.
7. assert in a verifier. python -O deletes it silently. Use raise.
8. Warning where the code's own assumption breaks. e.g. a 128-lane pack fed an HD that is not a multiple of 128 — warning + continue writes a wrong table.
9. Re-tokenization in loops. A pool cycled 40× re-tokenizing each fragment per pass; and a manifest recomputing a total by re-encoding the whole corpus. Cache counts once, accumulate during generation.
10. Substring tests standing in for key tests. "content" in line before json.loads(line)["content"] crashes on any list-shaped .json.
11. Path guards compared as strings. A/../A/x defeats an equality check; add realpath -m on both sides.
12. Shellcheck's honest false positive: single-quoted $CALIB passed to a converter that expands it from the container env is correct (SC2016). Say so in the PR; do not "fix" it.

Procedure

1. Inventory: wc -l the scripts + recipes; read each completely (ranges, not whole files, but every line). Recipes carry the numbers scripts assert.
2. Read the tests that pin the argv (tests/qwen38_reference.bats's SHAPE fixture, tests/qwen38_campaign.bats) — they are the contract you must not break, and the first place a stale number hides.
3. Mechanical gates:

   python3 -m py_compile scripts/*.py
   bash -n scripts/<each>.sh
   shellcheck -S warning scripts/<each>.sh      # never pass .py to it
   ./tests/run.sh
   

4. Fix the real ones only; keep diffs surgical; re-run the suite after the batch.
5. Smoke the changed path for real — smoke beats the suite:

   # run a local script on the server's data via the corpus venv, no checkout needed
   ssh dankserver '/home/dank/venvs/radcorpus/bin/python - --corpus <corpus> --out /tmp/<x> --force' \
     < scripts/rechunk-kld-corpus.py
   

   Import-level helpers can be exercised with importlib.util.spec_from_file_location (e.g. a new tap_bytes(): empty→0, one file→its size, absent→0).
6. Stacked PRs: a review commit goes on its own branch off the open one; a diff that includes the parent commit is a stacked PR, not a mistake — say which PR merges first in the body.

Report

One table of defect → cost → fix. Name the failure the defect produced in practice (hours wasted, wrong number), because that is what makes it reviewable.