Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
37 lines
4.3 KiB
Markdown
37 lines
4.3 KiB
Markdown
---
|
||
name: freetoken-hf-model-swap
|
||
description: "Swap the FreeToken model behind frank-default-small on Dankserver: preflight the HF quant format against the loader matrix BEFORE downloading, then download, repoint MODEL_DIR, deploy, and verify with a REAL completion (health 200 is not enough)."
|
||
---
|
||
|
||
# FreeToken HF Model Swap (frank-default-small, Dankserver)
|
||
|
||
Swap the model behind `frank-default-small` (FreeToken v0.1.2, Dankserver, RTX 3090) by downloading a HF safetensors checkpoint into a model dir, repointing `MODEL_DIR` in `scripts/frank-default-small.env`, and deploying. Verify with a REAL completion — health 200 is NOT enough.
|
||
|
||
## Preflight — verify quant format BEFORE downloading
|
||
|
||
FreeToken v0.1.2 loads ONLY modelopt-style quant configs for `qwen3_5_moe`. Every compressed-tensors FP8 checkpoint fails (live-verified 2026-09-01: two candidates, 72 GB downloaded, both unloadable). Check before any multi-GB download:
|
||
|
||
```bash
|
||
curl -sL https://huggingface.co/<org>/<repo>/raw/main/config.json | \
|
||
python3 -c "import json,sys; c=json.load(sys.stdin); q=c.get('quantization_config',{}); print(q.get('quant_method'), q.get('weight_block_size'), list((q.get('weight_scale') or ['?','?'])) if isinstance(q.get('weight_scale'),list) else q.get('weight_scale_inv') is not None)"
|
||
```
|
||
|
||
Decision matrix:
|
||
|
||
- `quant_method: "fp8"` + `weight_block_size: [128,128]` + `weight_scale_inv` → ✅ modelopt block-fp8 (the Huihui/official-Qwen format; only proven FP8 path)
|
||
- `quant_method: "compressed-tensors"` → ❌ FAILS. Per-channel aborts at expert-source resolution (`Missing MoE expert source layers: {'gate_up'}` when expert tensor naming is nonstandard); block-fp8 128×128 aborts in `_dequant_fp8_weight` (multiplies `weight × weight_scale` directly; block scales don't broadcast → shape RuntimeError)
|
||
- No quant config (BF16) → loads but 35B = ~70 GB weights >> 62 GB system RAM → sustained swap-thrash that locks out SSH (2026-09-01 outage). Never deploy BF16 for a 35B-class model on this box; a clean preflight compares disk size × precision against free RAM + 24 GB VRAM headroom.
|
||
- Also check: architecture `qwen3_5_moe`, expert tensor naming `mlp.experts.{N}.gate_proj/up_proj/down_proj` (read `model.safetensors.index.json` weight_map on HF if unsure — nonstandard prefixes defeat the MoE expert-source provider), config.json ships the chat template you want (FreeToken reads it from the checkpoint), disk space ≥ checkpoint size.
|
||
|
||
If the checkpoint fails the matrix, STOP — do not download, do not touch the running service. Find a compliant quant or use a different runtime.
|
||
|
||
## Swap steps
|
||
|
||
1. Download into `/home/dank/models/<name>/` via `snapshot_download` with the freetoken venv python (background the SSH call; poll until file count matches the index).
|
||
2. Edit `scripts/frank-default-small.env`: `MODEL_DIR`, description line (name, quant, arch, disk size, host, date), header comment. Keep F84 pool numbers unless the weight mass or KV dtype changed (FP8 35B ≈ 35 GB: `MODEL_CACHE_SIZE=2048`, `NUM_TOKENS=453005`, reasoning `on`; BF16 66 GB fits nothing — see preflight).
|
||
3. Run `./tests/run.sh`, commit, push to Gitea origin (git-only flow; never edit on the host).
|
||
4. `./scripts/deploy.sh dankserver --service frank-default-small`. The LLM_WAIT window is up to 20 min; load of ~35 GB FP8 is minutes warm.
|
||
5. **Verify for real** — the deploy health gate has reported "healthy" while a doomed worker was still starting (twice). Check journal for Uvicorn startup complete, then `curl /v1/models` (root = your model dir, ctx 262144) and a REAL completion: reasoning must parse into `reasoning_content` and content must answer correctly with a natural `finish_reason: stop`. Give reasoning models enough `max_tokens` to finish thinking (a tight budget yields empty content + `finish: length` — that is correct behavior, not a failure).
|
||
6. If load fails: journal shows the exact loader error, service exits cleanly (no swap-thrash), `Restart=on-failure` re-loop is harmless. Revert env via `git checkout <last-good-commit> -- <three files>`, redeploy, verify.
|
||
7. Update the runbook (`serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` quant matrix + history), `log.md`, and AGENTS.md fleet table; stage superseded model dirs to `~/.AGENTTRASH/<date>-topic/` on the host and list sizes for the user — never `rm`.
|