From fe34f044544dfa81493f517e6f971b47655120e4 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 1 Sep 2026 14:11:39 -0700 Subject: [PATCH] docs: record FreeToken compressed-tensors FP8 loader failures; Huihui restored Ornith per-channel aborts in expert-source resolution (nonstandard tensor naming); HauhauCS block-fp8 128x128 aborts in _dequant_fp8_weight (shape mismatch). modelopt block-fp8 is the only FP8 path. Log entry + runbook quant matrix updated; dead downloads staged to host trash. --- .../LLM/freetoken-frank-default-small.md | 19 +++++++++++- serverAppConfigs-docs/log.md | 30 +++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-) diff --git a/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md b/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md index 212ab61..8a29d15 100644 --- a/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md +++ b/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md @@ -19,7 +19,9 @@ The `frank-default-small` service on Dankserver is served by expert weights in host RAM with an LRU GPU cache. Model history on this runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) → `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30, -same format and pool budget). The engine itself replaced the llama.cpp +same format and pool budget; re-landed 2026-09-01 after two +compressed-tensors FP8 candidates failed to load — see Quantization +Compatibility). The engine itself replaced the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment. ## Model @@ -105,9 +107,24 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`: | NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ | | Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ | | Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug | +| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug | | Quark W8A8 INT8 | per-channel scales | ❌ loader bug | | GGUF | only `gemma4` arch | ❌ for qwen3_5_moe | +**2026-09-01 live verification (Dankserver):** both compressed-tensors +FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB +fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8` +(per-channel) aborts in expert-source resolution: +`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its +expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/ +down_proj` naming the MoE provider scans for. +`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block +128×128, standard naming) resolves experts but aborts in dequant: +`RuntimeError: The size of tensor a (2048) must match the size of +tensor b (...)` in `_dequant_fp8_weight`, which multiplies +`weight × weight_scale` directly instead of expanding the block scales. +modelopt block-fp8 remains the only FP8 path that loads. + ## Files | File | Purpose | diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 193370d..bd3928d 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,36 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-01 — frank-default-small: two compressed-tensors FP8 swaps fail to load; Huihui restored +- `/scripts/frank-default-small.env`, `/scripts/serve-frank-small.sh`, + `/scripts/frank-default-small.service` (`af6cd72` → `34ab6b4` → reverted + in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both + live-verified as loader failures and reverted the same day to the Huihui + FP8 checkpoint: + - `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel): + `ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the + checkpoint's expert tensors do not follow the + `mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider + scans for. + - `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` + (compressed-tensors block-fp8 128×128, standard naming): experts + resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale` + directly — block scales do not broadcast → + `RuntimeError: The size of tensor a (2048) must match ...`. +- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no + outage); service was back on Huihui within the hour and re-verified with + a real completion (reasoning parses; `17*23 → 391`, natural stop). +- `/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md` — quant + matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both + compressed-tensors shapes documented with their exact errors. +- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported + "healthy after approximately 6s" for BOTH failed loads — the health gate + passes while the doomed worker is still starting. A model swap is only + proven by the journal (Uvicorn startup) + a real completion. +- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8` + (37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8` + (35 GB). + ## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated - **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,