Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
41 lines
4.3 KiB
Markdown
41 lines
4.3 KiB
Markdown
---
|
|
name: hf-large-artifact-publication
|
|
description: "Publish a large quantized artifact + model card (and an optional anonymized corpus dataset) to HF Hub from dankserver: staging set, HF-style card spine, upload gates, transient-unit launcher, sentinel watching, card-metadata and scrub laws."
|
|
---
|
|
|
|
# HF large-artifact publication (dankserver)
|
|
|
|
Publish a big model artifact (100+ GiB) + card, and optionally an anonymized corpus dataset, from dankserver under `frank8465`.
|
|
|
|
## Staging
|
|
- Dev box `~/Projects/hf-swift15-upload/` (author card + scripts here), `scp -r` synced to `/home/dank/hf-swift15-upload/` on dankserver. Keep staging + logs + `.rc` sentinels after publication as receipts.
|
|
- Payload set: artifact, `README.md` card, `scripts/` (recipe/drivers/probes), `artifacts/` (gate JSONs, per-position `.rows`, manifests, run logs).
|
|
- Secret-scan the whole payload before arming: credential regexes (hf_/ghp_/pat, private keys, Bearer values, AKIA/sk-), LAN IPs, hostnames, domains, emails. Verdict each hit — variable *references* and upstream test fixtures are not leaks; check real values in context.
|
|
|
|
## Card law (HF validator)
|
|
- Custom licenses MUST be `license: other` + `license_name:` + `license_link:` (mirror the base model's card trio — fetch `https://huggingface.co/api/models/<base>` and read `cardData`). A raw non-SPDX `license:` value raises `ValueError: Invalid metadata in README.md` inside `upload_file` — AFTER all local gates have run.
|
|
- Card spine: what-the-format-is + engine pin digest, recipe table, quality gates (KL/top-1/PPL + per-position data shipped), proof hardware, exact `podman run` line, license chain, manifest table with the artifact's sha256 receipt, "In plain English:" block per section.
|
|
- Quote the SHIPPED filename (build name), not a prettified one — a hub-side rename of a 100 GiB LFS file costs a full re-upload.
|
|
|
|
## Upload mechanics
|
|
- venv `/home/dank/venvs/radcorpus` (huggingface_hub 1.28 + hf_xet); `HF_XET_HIGH_PERFORMANCE=1`; token from `~/.cache/huggingface/token` (see defuse-shell-rc-secrets pattern: `~/.config/secrets/<name>` 600 + rc loader, mirrored to the cache file). NEVER token in argv/chat.
|
|
- Script gates, in order: sha256 of live artifact vs recorded digest (`gate 1 : MATCH`), `create_repo(..., private=False, exist_ok=True)`, small files first, artifact last, post-upload census — **compare remote sizes to local and FAIL the RC on any mismatch** (this caught a stale card mid-campaign).
|
|
- hub 1.28 API traps: `upload_file`/`create_commit` kwargs are keyword-only; no `rename_file`, `hf_hub_url`, `HfApi.commit_api` (use `CommitOperationAdd`/`CommitOperationDelete` from `huggingface_hub` top-level if needed); `whoami` hides `canWrite` for legacy tokens — probe write scope by `create_repo(exist_ok=True)` on the real target.
|
|
- zsh-over-ssh: never `echo ===` (triggers `=cmd` expansion); quote it.
|
|
|
|
## Launcher + watchers
|
|
`setsid nohup` over ssh dies at session teardown. Use a transient user unit:
|
|
```
|
|
systemd-run --user --collect --unit=hf-upload -p SendSIGKILL=no \
|
|
-p StandardOutput=append:/home/dank/logs/hf-upload.log \
|
|
-p StandardError=append:/home/dank/logs/hf-upload.log \
|
|
sh -c "cd <staging> && <venv>/bin/python -u <uploader>.py --go; echo \$? > /home/dank/logs/hf-upload.rc"
|
|
```
|
|
Sentinel: background ssh loop polling the `.rc` file (60 s period, bounded), tails the log on appearance. Progress bars are `\r`-fed — `tr '\r' '\n' | tail` to read them.
|
|
|
|
## Corpus dataset variant
|
|
- License audit first: read `urls.txt`/manifest, fetch each source card (`license` in cardData). Share-alike sources (CC BY-SA) → publish derived corpus CC BY-SA with attribution; model weights unaffected.
|
|
- Fleet-harvested slices = agent-session transcripts: scrub with deterministic substring-class rules (NOT word-boundary: catches `dankclaw_default`, `FRANKBOOK_SVCS`, `franksdank/alloy` image namespace): LAN IPs→10.x, hostnames→server-a/b/client-c, domain→example.com (catch `\.`-escaped forms + bare brand), `/home/<user>`, emails→user@example.com. Residual probes run on OUTPUT; whitelist self-replacements + audited fixture noise.
|
|
- Publish original hashes, disclose non-byte-identity, cross-link model card ↔ dataset card, verify via `raw/main/README.md` fetches.
|
|
- Dataset repo needs dataset-write scope (fine-grained token; access tokens 403). `repo_type="dataset"` everywhere; 45 MB rides in ~60 s.
|