frank-default-small: retarget to lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8

Ornith-1.5-35B-A3B-FP8 failed to load on FreeToken v0.1.2: its
compressed-tensors checkpoint uses a nonstandard expert tensor layout
(FreeToken: 'Missing MoE expert source layers: gate_up').  Retarget to
the HauhauCS Aggressive FP8 finetune of the SAME Qwen3.6-35B-A3B arch
Huihui used, which carries compressed-tensors BLOCK-fp8 (128x128) with
the standard per-expert naming (mlp.experts.N.gate_proj/up_proj/down_proj
+ .weight_scale) that FreeToken's MoE loader reads.

- MODEL_DIR -> /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8
- use the checkpoint's own chat_template.jinja (drop the Sharp override;
  forcing a template the finetune wasn't trained on risks format drift)
- pool plan unchanged (same ~35 GB FP8 mass / qwen3_5_moe arch)
This commit is contained in:
Frank committed 2026-09-01 13:56:22 -07:00
1 parent a14a70a404
commit 34ab6b4d0c
3 files changed
+42 -44

No files matched your search

+12 -11
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Serve ornith-ai/Ornith-1.5-35B-A3B-FP8 as frank-default-small via
# FreeToken on Dankserver (RTX 3090, 62 GB RAM).
# Serve lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 as
# frank-default-small via FreeToken on Dankserver (RTX 3090, 62 GB RAM).
#
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
# - Experts live in host RAM; an LRU cache of expert slots is resident
@@ -10,14 +10,15 @@
# hold constant-size per-sequence state (mamba slots).
# - OpenAI-compatible API (/v1/chat/completions, /v1/models).
#
# Model: ornith-ai/Ornith-1.5-35B-A3B-FP8 — Ornith-1.5 (self-improvement
# RL-trained coding/agentic model over Qwen3.5 + Gemma4), 35B total / 3B
# active, in compressed-tensors per-channel FP8 (float-quantized, dynamic
# token activations). Same qwen3_5_moe arch and ~35 GB FP8 mass as the
# Huihui FP8 it replaces: 40 layers (10 full-attention + 30 GDN), 256
# experts / 8 used, 2 KV heads × 128 dim, native 262144 context. FreeToken
# loads a Qwen-Sharp chat template over it (see CHAT_TEMPLATE_FILE in the
# env file); the checkpoint's own stock template mis-serializes reasoning.
# Model: lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 —
# HauhauCS "Aggressive" uncensored finetune of Qwen3.6-35B-A3B in
# compressed-tensors block-fp8 (e4m3, dynamic group-128 activations,
# 128×128 weight blocks) with the standard per-expert tensor layout
# (mlp.experts.{N}.gate_proj/up_proj/down_proj + .weight_scale). Same
# qwen3_5_moe arch and ~35 GB FP8 mass as the Huihui FP8 it replaces: 40
# layers (10 full-attention + 30 GDN), 256 experts / 8 used, 2 KV heads ×
# 128 dim, native 262144 context. FreeToken reads the checkpoint's own
# chat_template.jinja from the model directory (no template override).
#
# Pool sizing: the GPU budget is split MoE-first between expert residency
# and KV pages. --moe-cache-size pins the number of resident expert
@@ -39,7 +40,7 @@ if [[ -f "$ENV_FILE" ]]; then
fi
# --- defaults (used when the env file does not set them) -----------------
: "${MODEL_DIR:=/home/dank/models/ornith-1.5-35b-a3b-fp8}"
: "${MODEL_DIR:=/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8}"
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"