command-center/agent/managed-skills/freetoken-hf-model-swap/SKILL.md

name, description

name	description
freetoken-hf-model-swap	Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough).

FreeToken HF Model Swap (frank-default-small, Dankserver)

Swap the model behind frank-default-small (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing MODEL_DIR in scripts/frank-default-small.env, and deploying. Verify with a REAL completion — health 200 is NOT enough.

Preflight — verify quant format BEFORE downloading

FreeToken v0.1.2 loads ONLY modelopt-style quant configs for qwen3_5_moe. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download:

curl -sL https://huggingface.co/<org>/<repo>/raw/main/config.json | \
  python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)"

Decision matrix:

• quant_method: "fp8" + weight_block_size: [128,128] + weight_scale_inv → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path)
• quant_method: "compressed-tensors" → ❌ FAILS. Per-channel aborts at expert-source resolution (Missing MoE expert source layers: {'gate_up'} when expert tensor naming is nonstandard); block-fp8 128×128 aborts in _dequant_fp8_weight (multiplies weight × weight_scale directly; block scales don't broadcast → shape RuntimeError)
• No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom.
• Also check: architecture qwen3_5_moe, expert tensor naming mlp.experts.{N}.gate_proj/up_proj/down_proj (read model.safetensors.index.json weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size.

If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime.

Swap steps

1. Download into /home/dank/models/<name>/ via snapshot_download with the freetoken venv python (background the SSH call; poll until file count matches the index).
2. Edit scripts/frank-default-small.env: MODEL_DIR, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: MODEL_CACHE_SIZE=2048, NUM_TOKENS=453005, reasoning on; BF16 66 GB fits nothing — see preflight).
3. Run ./tests/run.sh, commit, push to Gitea origin (git-only flow; never edit on the host).
4. ./scripts/deploy.sh dankserver --service frank-default-small. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm.
5. Verify for real — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then curl /v1/models (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into reasoning_content and content must answer correctly with a natural finish_reason: stop. Give reasoning models enough max_tokens to finish thinking (a tight budget yields empty content + finish: length — that is correct behavior, not a failure).
6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), Restart=on-failure re-loop is harmless. Revert env via git checkout <last-good-commit> -- <three files>, redeploy, verify.
7. Update the runbook (serverAppConfigs-docs/LLM/freetoken-frank-default-small.md quant matrix + history), log.md, and AGENTS.md fleet table; stage superseded model dirs to ~/.AGENTTRASH/<date>-topic/ on the host and list sizes for the user — never rm.