docs: record FreeToken compressed-tensors FP8 loader failures; Huihui restored
Ornith per-channel aborts in expert-source resolution (nonstandard tensor naming); HauhauCS block-fp8 128x128 aborts in _dequant_fp8_weight (shape mismatch). modelopt block-fp8 is the only FP8 path. Log entry + runbook quant matrix updated; dead downloads staged to host trash.
This commit is contained in:
1 parent
949f68e2b5
commit
fe34f04454
2 files changed
+48
-1
No files matched your search
@@ -19,7 +19,9 @@ The `frank-default-small` service on Dankserver is served by
|
||||
expert weights in host RAM with an LRU GPU cache. Model history on this
|
||||
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
|
||||
`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
|
||||
same format and pool budget). The engine itself replaced the llama.cpp
|
||||
same format and pool budget; re-landed 2026-09-01 after two
|
||||
compressed-tensors FP8 candidates failed to load — see Quantization
|
||||
Compatibility). The engine itself replaced the llama.cpp
|
||||
Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
|
||||
|
||||
## Model
|
||||
@@ -105,9 +107,24 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
|
||||
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
|
||||
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
|
||||
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
|
||||
| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
|
||||
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
|
||||
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
|
||||
|
||||
**2026-09-01 live verification (Dankserver):** both compressed-tensors
|
||||
FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
|
||||
fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
|
||||
(per-channel) aborts in expert-source resolution:
|
||||
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
|
||||
expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
|
||||
down_proj` naming the MoE provider scans for.
|
||||
`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
|
||||
128×128, standard naming) resolves experts but aborts in dequant:
|
||||
`RuntimeError: The size of tensor a (2048) must match the size of
|
||||
tensor b (...)` in `_dequant_fp8_weight`, which multiplies
|
||||
`weight × weight_scale` directly instead of expanding the block scales.
|
||||
modelopt block-fp8 remains the only FP8 path that loads.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|
||||
Reference in new issue
Block a user