frank-default-small: swap to batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken
- same qwen3_5_moe arch and native block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert layout) as the official Qwen FP8 checkpoint: identical FreeToken loader path, pool numbers unchanged (2048 MoE slots / 453005 KV tokens / 4 requests) - checkpoint downloaded to /home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8 (26 shards, ~35G); official checkpoint kept for rollback - ships froggeric fixed Qwen3.6 chat template, read from checkpoint - also refresh both .service Descriptions (large still said STRIX_LEAN)
This commit is contained in:
1 parent
3066d810ee
commit
aa9c1d065a
4 files changed
+29
-20
No files matched your search
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve Qwen/Qwen3.6-35B-A3B-FP8 as frank-default-small via FreeToken
|
||||
# on Dankserver (RTX 3090, 60 GB RAM).
|
||||
# Serve batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as
|
||||
# frank-default-small via FreeToken on Dankserver (RTX 3090, 60 GB RAM).
|
||||
#
|
||||
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
|
||||
# - Experts live in host RAM; an LRU cache of expert slots is resident
|
||||
@@ -10,10 +10,14 @@
|
||||
# hold constant-size per-sequence state (mamba slots).
|
||||
# - OpenAI-compatible API (/v1/chat/completions, /v1/models).
|
||||
#
|
||||
# Model: Qwen/Qwen3.6-35B-A3B-FP8 (safetensors, modelopt block-fp8,
|
||||
# 128x128 weight_block_size). Architecture: qwen3_5_moe — 40 layers,
|
||||
# full_attention_interval=4 (10 attention + 30 GDN), 256 experts / 8 used,
|
||||
# 2 KV heads × 256 dim. 35 GB on disk.
|
||||
# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 — Huihui
|
||||
# abliterated finetune of Qwen3.6-35B-A3B in the same native modelopt
|
||||
# block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert
|
||||
# layout) as the official FP8 checkpoint this service ran before; the
|
||||
# FreeToken loader path is unchanged. 40 layers (10 attention + 30
|
||||
# GDN), 256 experts / 8 used, 2 KV heads × 256 dim. ~35 GB on disk.
|
||||
# The checkpoint ships the fixed Qwen3.6 chat template; FreeToken reads
|
||||
# it from the model directory.
|
||||
#
|
||||
# Pool sizing: the GPU budget is split MoE-first between expert residency
|
||||
# and KV pages. --moe-cache-size pins the number of resident expert
|
||||
@@ -35,7 +39,7 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -----------------
|
||||
: "${MODEL_DIR:=/home/dank/models/qwen36-35b-a3b-fp8}"
|
||||
: "${MODEL_DIR:=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8}"
|
||||
: "${ALIAS:=frank-default-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
|
||||
Reference in new issue
Block a user