frank-default-small: revert to Huihui FP8 — FreeToken v0.1.2 cannot load compressed-tensors FP8
Two live-verified loader failures on Dankserver (2026-09-01), each a
clean exit at load (no swap-thrash, 37 GB fits RAM):
1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8):
ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]}
— nonstandard expert tensor naming, provider finds no expert sources.
2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
(compressed-tensors BLOCK-fp8 128x128, standard per-expert naming):
RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...) in _dequant_fp8_weight — the dequant path multiplies
weight * weight_scale directly and cannot expand compressed-tensors
block scales.
Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090):
modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv) -> LOADS
compressed-tensors per-channel FP8 -> fails
compressed-tensors block-fp8 128x128 -> fails
batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint;
restoring it until a modelopt/NVFP4 requant of a wanted model exists or
FreeToken learns compressed-tensors.
This commit is contained in:
1 parent
272ef6f0b0
commit
4d5bea42e0
3 files changed
+31
-47
No files matched your search
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=FreeToken Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small)
|
||||
Description=FreeToken Huihui-Qwen3.6-35B-A3B-abliterated-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
Reference in new issue
Block a user