78 lines
3.7 KiB
Markdown
78 lines
3.7 KiB
Markdown
---
|
|
name: script-refactor-parity-vs-previous-revision
|
|
description: "Prove a refactor of sibling data-pipeline scripts is behavior-identical: run the pre-refactor revision from git HEAD beside the new one on the host that owns the venv/data, compare parsed records rather than bytes, and bootstrap the shared module so file, runpy and container paths all import it."
|
|
---
|
|
|
|
Refactoring a pipeline of sibling scripts (corpus builders, re-chunkers, pushers) whose only
|
|
tests are smokes. The test suite cannot see a moved record boundary, so parity has to be
|
|
*measured against the previous revision*.
|
|
|
|
## 1. Keep the old revision runnable
|
|
|
|
Never trust the diff. Materialize both sides where the data lives (the host with the venv,
|
|
tokenizer, parquet, corpus):
|
|
|
|
```sh
|
|
git show HEAD:scripts/<tool>.py > /tmp/qc-old/<tool>.py # pre-refactor
|
|
cp scripts/<tool>.py scripts/<newmodule>.py /tmp/qc-new/ # post, module beside it
|
|
```
|
|
|
|
`ssh host 'cat > /tmp/...'` with the bytes on stdin is enough; no repo on the target needed.
|
|
Run both with **identical seeds and flags**, into separate output dirs.
|
|
|
|
## 2. Compare records, not bytes — unless the escaping is unchanged
|
|
|
|
- If the refactor funnels writes through one helper, that helper's `ensure_ascii` choice moves
|
|
every non-ASCII character. A byte diff then reports a false regression while the rows are
|
|
identical.
|
|
- Prove it correctly: parse both sides and compare in order —
|
|
|
|
```python
|
|
a = [json.loads(l) for l in open(old) if l.strip()]
|
|
b = [json.loads(l) for l in open(new) if l.strip()]
|
|
print(a == b, hashlib.sha256(json.dumps(a, sort_keys=True).encode()).hexdigest()[:16])
|
|
```
|
|
|
|
- Where the refactor does not touch serialization (a tool that already wrote literal UTF-8),
|
|
demand the stronger proof: `cmp` + md5 equality of the artifact.
|
|
- Order matters: shuffled output with an unchanged seed proves the RNG draw sequence survived
|
|
the hoist of nested closures. Compare the *sorted* file only as a second check — permutation
|
|
equality alone hides a reordered `rng.choice`.
|
|
- Hashing the record set with `sort_keys=True` also catches dict-key drift in manifests.
|
|
|
|
## 3. Sibling-module bootstrap that survives every invocation
|
|
|
|
A new shared module must import when the tool is run three ways: `python3 scripts/tool.py`,
|
|
`runpy.run_path('/work/tool.py')` from a container mount (`-v scripts:/work:ro`), and from any
|
|
cwd. Standard preamble:
|
|
|
|
```python
|
|
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
|
|
if _HERE not in sys.path:
|
|
sys.path.insert(0, _HERE)
|
|
import <module> # noqa: E402 - sibling module, same directory
|
|
```
|
|
|
|
Piping the script through `python3 -` has no `__file__` and defeats this — run tools by path and
|
|
say so in the module docstring. Verify both forms explicitly before claiming it works.
|
|
|
|
## 4. Unit-test the refusals without the heavy dependency
|
|
|
|
Hoist validation into a function so it can be driven with a stub, in-process:
|
|
|
|
```python
|
|
spec = importlib.util.spec_from_file_location("<module>", ".../<module>.py")
|
|
m = importlib.util.module_from_spec(spec); sys.modules["<module>"] = m; spec.loader.exec_module(m)
|
|
```
|
|
|
|
Registering the module in `sys.modules` under its real name *before* loading the dependent
|
|
script is required, or the script's own `import <module>` re-imports it and your stub is invisible.
|
|
A stub object with `encode`/`decode` that mangles ids past a threshold exercises a
|
|
"token identity" check in one page of output, no tokenizer file needed.
|
|
|
|
## 5. Gate order
|
|
|
|
`py_compile` every touched file → run the tool's `--help` through both invocation forms →
|
|
old-vs-new parity on the host → suite → commit. A suite that enumerates shell seams will not
|
|
notice a corpus-tool regression; the parity run is the only evidence there.
|