From 4d5bea42e05868fd003494a2c38030181c3b56d1 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 1 Sep 2026 14:02:27 -0700 Subject: [PATCH] =?UTF-8?q?frank-default-small:=20revert=20to=20Huihui=20F?= =?UTF-8?q?P8=20=E2=80=94=20FreeToken=20v0.1.2=20cannot=20load=20compresse?= =?UTF-8?q?d-tensors=20FP8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two live-verified loader failures on Dankserver (2026-09-01), each a clean exit at load (no swap-thrash, 37 GB fits RAM): 1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8): ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]} — nonstandard expert tensor naming, provider finds no expert sources. 2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 (compressed-tensors BLOCK-fp8 128x128, standard per-expert naming): RuntimeError: The size of tensor a (2048) must match the size of tensor b (...) in _dequant_fp8_weight — the dequant path multiplies weight * weight_scale directly and cannot expand compressed-tensors block scales. Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090): modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv) -> LOADS compressed-tensors per-channel FP8 -> fails compressed-tensors block-fp8 128x128 -> fails batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint; restoring it until a modelopt/NVFP4 requant of a wanted model exists or FreeToken learns compressed-tensors. --- scripts/frank-default-small.env | 53 +++++++++++------------------ scripts/frank-default-small.service | 2 +- scripts/serve-frank-small.sh | 23 ++++++------- 3 files changed, 31 insertions(+), 47 deletions(-) diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 7a00194..e0dc179 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -1,45 +1,30 @@ -# Tunables for frank-default-small (lamianlbe/Qwen3.6-35B-A3B-Uncensored- -# HauhauCS-Aggressive-fp8, qwen3_5_moe hybrid MoE, Dankserver, RTX 3090). +# Tunables for frank-default-small (batsclamp/Huihui-Qwen3.6-35B-A3B- +# abliterated-FP8, qwen3_5_moe hybrid MoE, Dankserver, RTX 3090). # # Runtime: FreeToken v0.1.2 (edge-native MoE serving engine). # Installed via uv into /home/dank/freetoken/.venv (CPython 3.12). # FreeToken keeps expert weights in host RAM with an LRU GPU cache; -# KV is paged with radix prefix sharing. Only the full-attention -# layers page KV (~20 KiB/token); the GDN layers use constant-size +# KV is paged with radix prefix sharing. Only the 10 full-attention +# layers page KV (~20 KiB/token); the 30 GDN layers use constant-size # per-sequence state (mamba slot pool). # -# Model: lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 -# (HuggingFace safetensors, ~37.5 GB on disk). HauhauCS "Aggressive" -# uncensored finetune of Qwen3.6-35B-A3B — same qwen3_5_moe arch as the -# Huihui FP8 it replaces (40 layers: 10 full-attention + 30 GDN, 256 -# experts / 8 used, 2 KV heads × 128 dim, native 262144 context) and the -# same ~35 GB FP8 weight mass, so the FreeToken memory/pool plan carries -# unchanged. Quantized to compressed-tensors BLOCK-fp8 (e4m3, dynamic -# group-128 activations, 128×128 weight blocks) with the standard -# per-expert tensor layout (mlp.experts.{N}.gate_proj/up_proj/down_proj -# + .weight_scale) that FreeToken's MoE loader reads — the layout the -# Ornith compressed-tensors checkpoint lacked. Also ships a BF16 visual -# encoder + MTP module (both in the quant "ignore" list; text-only -# serving ignores them). The RTX 3090 has no FP8 tensor cores, so FP8 -# runs via upcast. Replaced batsclamp/Huihui-...-FP8 on 2026-09-01. -# -# NOTE (loader risk): compressed-tensors block-fp8 is not the modelopt -# block-fp8 the runbook lists as ✅ — it shares the block layout and the -# per-expert naming but carries a different quant_method. Load is -# verified live; 37.5 GB fits RAM so a failure is a clean exit, never a -# swap-thrash. -# -# Chat template: the checkpoint ships its own chat_template.jinja (the -# format this finetune was trained on); FreeToken reads it from the model -# directory — no override. The fleet Qwen-Sharp template is deliberately -# NOT forced here: applying a template the model was not trained on risks -# format drift on an uncensored finetune. To force Sharp, set -# CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja. +# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (HuggingFace +# safetensors). Abliterated (refusal-removed) Huihui finetune of +# Qwen3.6-35B-A3B, quantized to the SAME native format the previous +# official checkpoint used: modelopt block-fp8 e4m3, dynamic +# activations, 128x128 weight_block_size, bf16 weight_scale_inv; +# lm_head/embeds/norms/router gates/visual/MTP kept BF16. Un-fused +# per-expert layout matches the official Qwen FP8, so the FreeToken +# loader path is identical; ~35 GB on disk. Ships the fixed Qwen3.6 +# chat template (froggeric/Qwen-Fixed-Chat-Templates) — FreeToken +# reads it from the checkpoint; no template override needed. +# Replaced Qwen/Qwen3.6-35B-A3B-FP8 on 2026-08-30; pool numbers +# unchanged — same arch, same weight mass. # # Pool sizing: GPU budget (~17.6 GiB after weights + fixed caches) is # split MoE-first. MOE_CACHE_SIZE pins resident expert slots; NUM_TOKENS # pins the KV pool. At 2048 slots / 453005 tokens the split is ~20% -# residency and ~8.6 GiB KV. Live resize via +# residency (2048/10240) and ~8.6 GiB KV. Live resize via # `ft ctl cache --kv N --moe M` can grow KV by shrinking MoE without # restart (the reverse requires a restart). # @@ -47,8 +32,8 @@ # The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # incident). This box is NVIDIA/CUDA, but the rule keeps the env portable. -# Model directory (safetensors checkpoint, ~37.5 GB, compressed-tensors block-fp8). -MODEL_DIR=/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8 +# Model directory (safetensors checkpoint, ~35 GB, block-fp8). +MODEL_DIR=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8 # FreeToken venv (uv-managed CPython 3.12, ft CLI, flashinfer, torch+cu130). FT_VENV=/home/dank/freetoken/.venv diff --git a/scripts/frank-default-small.service b/scripts/frank-default-small.service index 2229a12..fe4f791 100644 --- a/scripts/frank-default-small.service +++ b/scripts/frank-default-small.service @@ -1,5 +1,5 @@ [Unit] -Description=FreeToken Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small) +Description=FreeToken Huihui-Qwen3.6-35B-A3B-abliterated-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small) After=network-online.target Wants=network-online.target diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index cfece36..fd1b396 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# Serve lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 as -# frank-default-small via FreeToken on Dankserver (RTX 3090, 62 GB RAM). +# Serve batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as +# frank-default-small via FreeToken on Dankserver (RTX 3090, 60 GB RAM). # # Runtime: FreeToken v0.1.2 (edge-native MoE serving engine). # - Experts live in host RAM; an LRU cache of expert slots is resident @@ -10,15 +10,14 @@ # hold constant-size per-sequence state (mamba slots). # - OpenAI-compatible API (/v1/chat/completions, /v1/models). # -# Model: lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 — -# HauhauCS "Aggressive" uncensored finetune of Qwen3.6-35B-A3B in -# compressed-tensors block-fp8 (e4m3, dynamic group-128 activations, -# 128×128 weight blocks) with the standard per-expert tensor layout -# (mlp.experts.{N}.gate_proj/up_proj/down_proj + .weight_scale). Same -# qwen3_5_moe arch and ~35 GB FP8 mass as the Huihui FP8 it replaces: 40 -# layers (10 full-attention + 30 GDN), 256 experts / 8 used, 2 KV heads × -# 128 dim, native 262144 context. FreeToken reads the checkpoint's own -# chat_template.jinja from the model directory (no template override). +# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 — Huihui +# abliterated finetune of Qwen3.6-35B-A3B in the same native modelopt +# block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert +# layout) as the official FP8 checkpoint this service ran before; the +# FreeToken loader path is unchanged. 40 layers (10 attention + 30 +# GDN), 256 experts / 8 used, 2 KV heads × 256 dim. ~35 GB on disk. +# The checkpoint ships the fixed Qwen3.6 chat template; FreeToken reads +# it from the model directory. # # Pool sizing: the GPU budget is split MoE-first between expert residency # and KV pages. --moe-cache-size pins the number of resident expert @@ -40,7 +39,7 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ----------------- -: "${MODEL_DIR:=/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8}" +: "${MODEL_DIR:=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8}" : "${ALIAS:=frank-default-small}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}"