frank-default-small: swap to ornith-ai/Ornith-1.5-35B-A3B-FP8 + Qwen-Sharp chat template
- MODEL_DIR -> /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB, compressed-tensors per-channel FP8, same qwen3_5_moe arch/mass as Huihui so the pool plan carries unchanged) - CHAT_TEMPLATE_FILE -> scripts/chat-template.jinja (Qwen-Sharp, shared with frank-default-large); the checkpoint's stock template mis-serializes reasoning per its own model card - env/serve/unit headers updated; loader-risk + BF16-does-not-fit note
This commit is contained in:
1 parent
cdfaf48191
commit
af6cd72d02
3 files changed
+49
-31
No files matched your search
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as
|
||||
# frank-default-small via FreeToken on Dankserver (RTX 3090, 60 GB RAM).
|
||||
# Serve ornith-ai/Ornith-1.5-35B-A3B-FP8 as frank-default-small via
|
||||
# FreeToken on Dankserver (RTX 3090, 62 GB RAM).
|
||||
#
|
||||
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
|
||||
# - Experts live in host RAM; an LRU cache of expert slots is resident
|
||||
@@ -10,14 +10,14 @@
|
||||
# hold constant-size per-sequence state (mamba slots).
|
||||
# - OpenAI-compatible API (/v1/chat/completions, /v1/models).
|
||||
#
|
||||
# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 — Huihui
|
||||
# abliterated finetune of Qwen3.6-35B-A3B in the same native modelopt
|
||||
# block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert
|
||||
# layout) as the official FP8 checkpoint this service ran before; the
|
||||
# FreeToken loader path is unchanged. 40 layers (10 attention + 30
|
||||
# GDN), 256 experts / 8 used, 2 KV heads × 256 dim. ~35 GB on disk.
|
||||
# The checkpoint ships the fixed Qwen3.6 chat template; FreeToken reads
|
||||
# it from the model directory.
|
||||
# Model: ornith-ai/Ornith-1.5-35B-A3B-FP8 — Ornith-1.5 (self-improvement
|
||||
# RL-trained coding/agentic model over Qwen3.5 + Gemma4), 35B total / 3B
|
||||
# active, in compressed-tensors per-channel FP8 (float-quantized, dynamic
|
||||
# token activations). Same qwen3_5_moe arch and ~35 GB FP8 mass as the
|
||||
# Huihui FP8 it replaces: 40 layers (10 full-attention + 30 GDN), 256
|
||||
# experts / 8 used, 2 KV heads × 128 dim, native 262144 context. FreeToken
|
||||
# loads a Qwen-Sharp chat template over it (see CHAT_TEMPLATE_FILE in the
|
||||
# env file); the checkpoint's own stock template mis-serializes reasoning.
|
||||
#
|
||||
# Pool sizing: the GPU budget is split MoE-first between expert residency
|
||||
# and KV pages. --moe-cache-size pins the number of resident expert
|
||||
@@ -39,7 +39,7 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -----------------
|
||||
: "${MODEL_DIR:=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8}"
|
||||
: "${MODEL_DIR:=/home/dank/models/ornith-1.5-35b-a3b-fp8}"
|
||||
: "${ALIAS:=frank-default-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
|
||||
Reference in new issue
Block a user