command-center/agent/managed-skills/hf-large-artifact-publication/SKILL.md

name, description

name	description
hf-large-artifact-publication	Publish a large quantized artifact + model card (and an optional anonymized corpus dataset) to HF Hub from dankserver: staging set, HF-style card spine, upload gates, transient-unit launcher, sentinel watching, card-metadata and scrub laws.

HF large-artifact publication (dankserver)

Publish a big model artifact (100+ GiB) + card, and optionally an anonymized corpus dataset, from dankserver under frank8465.

Staging

• Dev box ~/Projects/hf-swift15-upload/ (author card + scripts here), scp -r synced to /home/dank/hf-swift15-upload/ on dankserver. Keep staging + logs + .rc sentinels after publication as receipts.
• Payload set: artifact, README.md card, scripts/ (recipe/drivers/probes), artifacts/ (gate JSONs, per-position .rows, manifests, run logs).
• Secret-scan the whole payload before arming: credential regexes (hf_/ghp_/pat, private keys, Bearer values, AKIA/sk-), LAN IPs, hostnames, domains, emails. Verdict each hit — variable references and upstream test fixtures are not leaks; check real values in context.

Card law (HF validator)

• Custom licenses MUST be license: other + license_name: + license_link: (mirror the base model's card trio — fetch https://huggingface.co/api/models/<base> and read cardData). A raw non-SPDX license: value raises ValueError: Invalid metadata in README.md inside upload_file — AFTER all local gates have run.
• Card spine: what-the-format-is + engine pin digest, recipe table, quality gates (KL/top-1/PPL + per-position data shipped), proof hardware, exact podman run line, license chain, manifest table with the artifact's sha256 receipt, "In plain English:" block per section.
• Quote the SHIPPED filename (build name), not a prettified one — a hub-side rename of a 100 GiB LFS file costs a full re-upload.

Upload mechanics

• venv /home/dank/venvs/radcorpus (huggingface_hub 1.28 + hf_xet); HF_XET_HIGH_PERFORMANCE=1; token from ~/.cache/huggingface/token (see defuse-shell-rc-secrets pattern: ~/.config/secrets/<name> 600 + rc loader, mirrored to the cache file). NEVER token in argv/chat.
• Script gates, in order: sha256 of live artifact vs recorded digest (gate 1 : MATCH), create_repo(..., private=False, exist_ok=True), small files first, artifact last, post-upload census — compare remote sizes to local and FAIL the RC on any mismatch (this caught a stale card mid-campaign).
• hub 1.28 API traps: upload_file/create_commit kwargs are keyword-only; no rename_file, hf_hub_url, HfApi.commit_api (use CommitOperationAdd/CommitOperationDelete from huggingface_hub top-level if needed); whoami hides canWrite for legacy tokens — probe write scope by create_repo(exist_ok=True) on the real target.
• zsh-over-ssh: never echo === (triggers =cmd expansion); quote it.

Launcher + watchers

setsid nohup over ssh dies at session teardown. Use a transient user unit:

systemd-run --user --collect --unit=hf-upload -p SendSIGKILL=no \
  -p StandardOutput=append:/home/dank/logs/hf-upload.log \
  -p StandardError=append:/home/dank/logs/hf-upload.log \
  sh -c "cd <staging> && <venv>/bin/python -u <uploader>.py --go; echo \$? > /home/dank/logs/hf-upload.rc"

Sentinel: background ssh loop polling the .rc file (60 s period, bounded), tails the log on appearance. Progress bars are \r-fed — tr '\r' '\n' | tail to read them.

Corpus dataset variant

• License audit first: read urls.txt/manifest, fetch each source card (license in cardData). Share-alike sources (CC BY-SA) → publish derived corpus CC BY-SA with attribution; model weights unaffected.
• Fleet-harvested slices = agent-session transcripts: scrub with deterministic substring-class rules (NOT word-boundary: catches dankclaw_default, FRANKBOOK_SVCS, franksdank/alloy image namespace): LAN IPs→10.x, hostnames→server-a/b/client-c, domain→example.com (catch \.-escaped forms + bare brand), /home/<user>, emails→user@example.com. Residual probes run on OUTPUT; whitelist self-replacements + audited fixture noise.
• Publish original hashes, disclose non-byte-identity, cross-link model card ↔ dataset card, verify via raw/main/README.md fetches.
• Dataset repo needs dataset-write scope (fine-grained token; access tokens 403). repo_type="dataset" everywhere; 45 MB rides in ~60 s.