frank-default-small: revert to Huihui FP8 — FreeToken v0.1.2 cannot load compressed-tensors FP8

Two live-verified loader failures on Dankserver (2026-09-01), each a
clean exit at load (no swap-thrash, 37 GB fits RAM):

1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8):
   ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]}
   — nonstandard expert tensor naming, provider finds no expert sources.

2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
   (compressed-tensors BLOCK-fp8 128x128, standard per-expert naming):
   RuntimeError: The size of tensor a (2048) must match the size of
   tensor b (...) in _dequant_fp8_weight — the dequant path multiplies
   weight * weight_scale directly and cannot expand compressed-tensors
   block scales.

Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090):
  modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv)  -> LOADS
  compressed-tensors per-channel FP8                          -> fails
  compressed-tensors block-fp8 128x128                        -> fails

batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint;
restoring it until a modelopt/NVFP4 requant of a wanted model exists or
FreeToken learns compressed-tensors.
This commit is contained in:
Frank committed 2026-09-01 14:02:27 -07:00
1 parent 272ef6f0b0
commit 4d5bea42e0
3 files changed
+31 -47

No files matched your search

+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=FreeToken Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small)
Description=FreeToken Huihui-Qwen3.6-35B-A3B-abliterated-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small)
After=network-online.target
Wants=network-online.target