frank-default-small: retarget to lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
Ornith-1.5-35B-A3B-FP8 failed to load on FreeToken v0.1.2: its compressed-tensors checkpoint uses a nonstandard expert tensor layout (FreeToken: 'Missing MoE expert source layers: gate_up'). Retarget to the HauhauCS Aggressive FP8 finetune of the SAME Qwen3.6-35B-A3B arch Huihui used, which carries compressed-tensors BLOCK-fp8 (128x128) with the standard per-expert naming (mlp.experts.N.gate_proj/up_proj/down_proj + .weight_scale) that FreeToken's MoE loader reads. - MODEL_DIR -> /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8 - use the checkpoint's own chat_template.jinja (drop the Sharp override; forcing a template the finetune wasn't trained on risks format drift) - pool plan unchanged (same ~35 GB FP8 mass / qwen3_5_moe arch)
This commit is contained in:
1 parent
a14a70a404
commit
34ab6b4d0c
3 files changed
+42
-44
No files matched your search
@@ -1,5 +1,5 @@
|
||||
# Tunables for frank-default-small (ornith-ai/Ornith-1.5-35B-A3B-FP8,
|
||||
# qwen3_5_moe hybrid MoE, Dankserver, RTX 3090).
|
||||
# Tunables for frank-default-small (lamianlbe/Qwen3.6-35B-A3B-Uncensored-
|
||||
# HauhauCS-Aggressive-fp8, qwen3_5_moe hybrid MoE, Dankserver, RTX 3090).
|
||||
#
|
||||
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
|
||||
# Installed via uv into /home/dank/freetoken/.venv (CPython 3.12).
|
||||
@@ -8,30 +8,33 @@
|
||||
# layers page KV (~20 KiB/token); the GDN layers use constant-size
|
||||
# per-sequence state (mamba slot pool).
|
||||
#
|
||||
# Model: ornith-ai/Ornith-1.5-35B-A3B-FP8 (HuggingFace safetensors,
|
||||
# compressed-tensors FP8, ~35 GB on disk). Ornith-1.5 is a
|
||||
# self-improvement RL-trained coding/agentic model built on Qwen3.5 +
|
||||
# Gemma4; ~35B total / ~3B active (A3B), same qwen3_5_moe arch and the
|
||||
# same ~35 GB FP8 weight mass as the Huihui FP8 it replaces, so the
|
||||
# FreeToken memory/pool plan carries unchanged. Native 262144 context.
|
||||
# Reasoning model: each assistant turn opens with a <think> block,
|
||||
# surfaced separately via the qwen3 reasoning parser (REASONING=on).
|
||||
# The RTX 3090 has no FP8 tensor cores, so FP8 runs via upcast (as
|
||||
# Huihui did). Replaced batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-
|
||||
# FP8 on 2026-09-01.
|
||||
# Model: lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
|
||||
# (HuggingFace safetensors, ~37.5 GB on disk). HauhauCS "Aggressive"
|
||||
# uncensored finetune of Qwen3.6-35B-A3B — same qwen3_5_moe arch as the
|
||||
# Huihui FP8 it replaces (40 layers: 10 full-attention + 30 GDN, 256
|
||||
# experts / 8 used, 2 KV heads × 128 dim, native 262144 context) and the
|
||||
# same ~35 GB FP8 weight mass, so the FreeToken memory/pool plan carries
|
||||
# unchanged. Quantized to compressed-tensors BLOCK-fp8 (e4m3, dynamic
|
||||
# group-128 activations, 128×128 weight blocks) with the standard
|
||||
# per-expert tensor layout (mlp.experts.{N}.gate_proj/up_proj/down_proj
|
||||
# + .weight_scale) that FreeToken's MoE loader reads — the layout the
|
||||
# Ornith compressed-tensors checkpoint lacked. Also ships a BF16 visual
|
||||
# encoder + MTP module (both in the quant "ignore" list; text-only
|
||||
# serving ignores them). The RTX 3090 has no FP8 tensor cores, so FP8
|
||||
# runs via upcast. Replaced batsclamp/Huihui-...-FP8 on 2026-09-01.
|
||||
#
|
||||
# RISK: this is compressed-tensors PER-CHANNEL FP8 (weights strategy
|
||||
# "channel"), the one FP8 layout the runbook flags as a FreeToken v0.1.2
|
||||
# loader bug. Load is being verified live; 37 GB fits RAM so a failure is
|
||||
# a clean crash-loop, not a swap-thrash. The BF16 sibling
|
||||
# (ornith-ai/Ornith-1.5-35B-A3B, ~70 GB) does NOT fit this 62 GB box —
|
||||
# never fall back to it (that is the 2026-09-01 Thomson outage mode).
|
||||
# NOTE (loader risk): compressed-tensors block-fp8 is not the modelopt
|
||||
# block-fp8 the runbook lists as ✅ — it shares the block layout and the
|
||||
# per-expert naming but carries a different quant_method. Load is
|
||||
# verified live; 37.5 GB fits RAM so a failure is a clean exit, never a
|
||||
# swap-thrash.
|
||||
#
|
||||
# Chat template: the checkpoint's stock Qwen template is WRONG for this
|
||||
# model (its own model card says so and ships a modified
|
||||
# chat_template.jinja for the agentic benchmarks). The fleet standard is
|
||||
# the Qwen-Sharp template (scripts/chat-template.jinja, repo-tracked and
|
||||
# shared with frank-default-large) — see CHAT_TEMPLATE_FILE below.
|
||||
# Chat template: the checkpoint ships its own chat_template.jinja (the
|
||||
# format this finetune was trained on); FreeToken reads it from the model
|
||||
# directory — no override. The fleet Qwen-Sharp template is deliberately
|
||||
# NOT forced here: applying a template the model was not trained on risks
|
||||
# format drift on an uncensored finetune. To force Sharp, set
|
||||
# CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja.
|
||||
#
|
||||
# Pool sizing: GPU budget (~17.6 GiB after weights + fixed caches) is
|
||||
# split MoE-first. MOE_CACHE_SIZE pins resident expert slots; NUM_TOKENS
|
||||
@@ -44,14 +47,8 @@
|
||||
# The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
|
||||
|
||||
# Model directory (safetensors checkpoint, ~35 GB, compressed-tensors FP8).
|
||||
MODEL_DIR=/home/dank/models/ornith-1.5-35b-a3b-fp8
|
||||
|
||||
# Chat template: the fleet Qwen-Sharp template (repo-tracked), shared with
|
||||
# frank-default-large. The checkpoint's stock template mis-serializes
|
||||
# reasoning for this model; the Sharp template renders the <think> trail and
|
||||
# the Qwen3.5-family tool-call shape correctly.
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
# Model directory (safetensors checkpoint, ~37.5 GB, compressed-tensors block-fp8).
|
||||
MODEL_DIR=/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8
|
||||
|
||||
# FreeToken venv (uv-managed CPython 3.12, ft CLI, flashinfer, torch+cu130).
|
||||
FT_VENV=/home/dank/freetoken/.venv
|
||||
|
||||
Reference in new issue
Block a user