docs: record FreeToken compressed-tensors FP8 loader failures; Huihui restored

Ornith per-channel aborts in expert-source resolution (nonstandard
tensor naming); HauhauCS block-fp8 128x128 aborts in _dequant_fp8_weight
(shape mismatch). modelopt block-fp8 is the only FP8 path. Log entry +
runbook quant matrix updated; dead downloads staged to host trash.
This commit is contained in:
Frank committed 2026-09-01 14:11:39 -07:00
1 parent 949f68e2b5
commit fe34f04454
2 files changed
+48 -1

No files matched your search

@@ -19,7 +19,9 @@ The `frank-default-small` service on Dankserver is served by
expert weights in host RAM with an LRU GPU cache. Model history on this expert weights in host RAM with an LRU GPU cache. Model history on this
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) → runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30, `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
same format and pool budget). The engine itself replaced the llama.cpp same format and pool budget; re-landed 2026-09-01 after two
compressed-tensors FP8 candidates failed to load — see Quantization
Compatibility). The engine itself replaced the llama.cpp
Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment. Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
## Model ## Model
@@ -105,9 +107,24 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ | | NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ | | Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug | | Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug | | Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe | | GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
**2026-09-01 live verification (Dankserver):** both compressed-tensors
FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
(per-channel) aborts in expert-source resolution:
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
down_proj` naming the MoE provider scans for.
`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
128×128, standard naming) resolves experts but aborts in dequant:
`RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...)` in `_dequant_fp8_weight`, which multiplies
`weight × weight_scale` directly instead of expanding the block scales.
modelopt block-fp8 remains the only FP8 path that loads.
## Files ## Files
| File | Purpose | | File | Purpose |
+30
View File
@@ -11,6 +11,36 @@ timestamp: 2026-09-14
# Bundle Change Log # Bundle Change Log
## 2026-09-01 — frank-default-small: two compressed-tensors FP8 swaps fail to load; Huihui restored
- `/scripts/frank-default-small.env`, `/scripts/serve-frank-small.sh`,
`/scripts/frank-default-small.service` (`af6cd72` → `34ab6b4` → reverted
in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both
live-verified as loader failures and reverted the same day to the Huihui
FP8 checkpoint:
- `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel):
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the
checkpoint's expert tensors do not follow the
`mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider
scans for.
- `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8`
(compressed-tensors block-fp8 128×128, standard naming): experts
resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale`
directly — block scales do not broadcast →
`RuntimeError: The size of tensor a (2048) must match ...`.
- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no
outage); service was back on Huihui within the hour and re-verified with
a real completion (reasoning parses; `17*23 → 391`, natural stop).
- `/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` — quant
matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both
compressed-tensors shapes documented with their exact errors.
- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported
"healthy after approximately 6s" for BOTH failed loads — the health gate
passes while the doomed worker is still starting. A model swap is only
proven by the journal (Uvicorn startup) + a real completion.
- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8`
(37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8`
(35 GB).
## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated ## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated
- **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every - **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every
bare compose-era media container (RomM trio, Jellyfin, Calibre-Web, bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,