frank-default-small: swap to batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken

- same qwen3_5_moe arch and native block-fp8 format (e4m3, 128x128
  weight_block_size, un-fused per-expert layout) as the official
  Qwen FP8 checkpoint: identical FreeToken loader path, pool numbers
  unchanged (2048 MoE slots / 453005 KV tokens / 4 requests)
- checkpoint downloaded to /home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8
  (26 shards, ~35G); official checkpoint kept for rollback
- ships froggeric fixed Qwen3.6 chat template, read from checkpoint
- also refresh both .service Descriptions (large still said STRIX_LEAN)
This commit is contained in:
Frank committed 2026-08-30 02:26:49 -07:00
1 parent 3066d810ee
commit aa9c1d065a
4 files changed
+29 -20

No files matched your search

+11 -7
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Serve Qwen/Qwen3.6-35B-A3B-FP8 as frank-default-small via FreeToken
# on Dankserver (RTX 3090, 60 GB RAM).
# Serve batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as
# frank-default-small via FreeToken on Dankserver (RTX 3090, 60 GB RAM).
#
# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
# - Experts live in host RAM; an LRU cache of expert slots is resident
@@ -10,10 +10,14 @@
# hold constant-size per-sequence state (mamba slots).
# - OpenAI-compatible API (/v1/chat/completions, /v1/models).
#
# Model: Qwen/Qwen3.6-35B-A3B-FP8 (safetensors, modelopt block-fp8,
# 128x128 weight_block_size). Architecture: qwen3_5_moe — 40 layers,
# full_attention_interval=4 (10 attention + 30 GDN), 256 experts / 8 used,
# 2 KV heads × 256 dim. 35 GB on disk.
# Model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 — Huihui
# abliterated finetune of Qwen3.6-35B-A3B in the same native modelopt
# block-fp8 format (e4m3, 128x128 weight_block_size, un-fused per-expert
# layout) as the official FP8 checkpoint this service ran before; the
# FreeToken loader path is unchanged. 40 layers (10 attention + 30
# GDN), 256 experts / 8 used, 2 KV heads × 256 dim. ~35 GB on disk.
# The checkpoint ships the fixed Qwen3.6 chat template; FreeToken reads
# it from the model directory.
#
# Pool sizing: the GPU budget is split MoE-first between expert residency
# and KV pages. --moe-cache-size pins the number of resident expert
@@ -35,7 +39,7 @@ if [[ -f "$ENV_FILE" ]]; then
fi
# --- defaults (used when the env file does not set them) -----------------
: "${MODEL_DIR:=/home/dank/models/qwen36-35b-a3b-fp8}"
: "${MODEL_DIR:=/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8}"
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"