serverAppConfigs/serverAppConfigs-docs/LLM/freetoken-frank-small.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	FreeToken frank-small Runbook	FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090).	freetoken dankserver frank-small qwen3.6 moe fp8	2026-08-30

FreeToken frank-small Runbook

The frank-small service on Dankserver is served by FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache. Model history on this runtime: official Qwen/Qwen3.6-35B-A3B-FP8 (since 2026-09-02) → batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (2026-08-30, same format and pool budget; re-landed 2026-09-01 after two compressed-tensors FP8 candidates failed to load — see Quantization Compatibility). The engine itself replaced the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.

Model

Field	Value
HuggingFace	batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8
Lineage	Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp)
Architecture	qwen3_5_moe (Qwen3_5MoeForConditionalGeneration)
Quant	native modelopt block-fp8 e4m3, dynamic activations, 128×128 weight_block_size, bf16 weight_scale_inv — byte-format identical to the official FP8 repo
BF16 kept	lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module
Experts	un-fused per-expert layout (mlp.experts.{0..255}.*) matching official Qwen FP8 — same FreeToken loader path
Size on disk	~35 GB (26 safetensors shards)
Path	/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8
Layers	40 (10 full-attention + 30 GDN/linear-attention)
Experts	256 per layer, 8 used per token
KV heads	2 × 256 dim
Context	262,144 (model max)
Chat template	froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in tokenizer_config.json; FreeToken reads it from the checkpoint — no override)

Important: this is the Huihui abliterated finetune (refusals removed) — not the base Qwen weights served before 2026-08-30, and not the earlier Tiel-Coder finetune. Expect a direct-answers assistant.

Runtime

Field	Value
Engine	FreeToken v0.1.2 (ft CLI)
Venv	/home/dank/freetoken/.venv (uv-managed CPython 3.12)
Dependencies	torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi
CUDA toolchain	pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86
API	OpenAI-compatible (/v1/chat/completions, /v1/models)

Pool Configuration

The GPU budget (~17.6 GiB after weights + fixed caches) is split MoE-first between expert residency and KV pages:

Pool	Size	VRAM	Notes
MoE cache	2,048 slots (20%)	6.0 GiB	LRU expert cache, out of 10,240 total
KV cache	453,005 tokens	8.6 GiB	~20 KiB/token (10 attn layers only)
Mamba state	24 slots	1.4 GiB	Per-sequence GDN state (caps concurrency)

Live Resize

KV and MoE can be re-partitioned without restart:

ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300

• Growing KV (shrinking MoE): works live.
• Shrinking KV (growing MoE): requires restart (old + new pools cannot coexist in VRAM during the swap → OOM).
• The fit check refuses overcommit: if the plan doesn't fit, the old cache is retained and serving continues.

Benchmark Results (2026-09-02, base FP8 checkpoint)

See freetoken-benchmark-sweep for the full bs1/bs3/bs4/bs6 sweep across KV pool sizes 8k–700k.

At the production config (20% residency, bs4):

Metric	Value
bs1 decode	~30 tok/s
bs4 aggregate	~87 tok/s
bs4 per-request	~22 tok/s
Prefill (4k)	~1,130 tok/s

Quantization Compatibility

FreeToken v0.1.2 supports these safetensor quants for qwen3_5_moe:

Format	Detection	Works
BF16 (no quant config)	default	✅ (experts too large for 60 GB RAM)
modelopt block-fp8 (128×128)	quant_method: "fp8" + weight_block_size	✅ (current)
NVFP4 (modelopt)	quant_algo: "NVFP4"	✅ (best for VRAM)
NVFP4 (compressed-tensors)	4-bit float, group_size=16, strategy=tensor_group	✅
Per-tensor FP8 (modelopt)	scalar weight_scale	✅
Per-channel FP8 (compressed-tensors)	weight_scale: [O, 1]	❌ loader bug
Block-fp8 128×128 (compressed-tensors)	quant_method: "compressed-tensors" + block_structure: [128, 128]	❌ dequant shape bug
Quark W8A8 INT8	per-channel scales	❌ loader bug
GGUF	only gemma4 arch	❌ for qwen3_5_moe

2026-09-01 live verification (Dankserver): both compressed-tensors FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB fits RAM — no swap-thrash). ornith-ai/Ornith-1.5-35B-A3B-FP8 (per-channel) aborts in expert-source resolution: ValueError: Missing MoE expert source layers: {'gate_up': ...} — its expert tensors do not follow the mlp.experts.{N}.gate_proj/up_proj/ down_proj naming the MoE provider scans for. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 (block 128×128, standard naming) resolves experts but aborts in dequant: RuntimeError: The size of tensor a (2048) must match the size of tensor b (...) in _dequant_fp8_weight, which multiplies weight × weight_scale directly instead of expanding the block scales. modelopt block-fp8 remains the only FP8 path that loads.

Files

File	Purpose
scripts/serve-frank-small.sh	Serve script (launches ft serve)
scripts/frank-small.env	Tunables (model path, pool sizes)
scripts/frank-small.service	Systemd unit

Limitations

• Model identity: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
• JIT compilation: first startup compiles flashinfer/tvm-ffi kernels (~5–15 min cold). Cached afterward in ~/.cache/flashinfer and ~/.cache/tvm-ffi.
• CUDA toolchain: driver-only install (no system CUDA toolkit). The venv provides pip-installed nvcc 13.2.86 for JIT compilation. All nvidia-cu13 packages must be at the same version.
• Env var collisions: never name tunables TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads these).

Operational quirks

• model field is required in /v1/chat/completions bodies — FreeToken rejects requests without it with a 422 (Field required), unlike llama.cpp's server which defaults to the loaded model. Send "model": "frank-small".
• Load on this swap took well under the old llama.cpp numbers: weights → experts (parallel build) → CUDA graph capture → Uvicorn in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a 282-token completion.

Citations

• FreeToken paper: arXiv 2608.16157
• FreeToken PyPI: freetoken[accel] v0.1.2
• HuggingFace model: batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8
• Upstream weights: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated, Qwen/Qwen3.6-35B-A3B-FP8
• Fixed chat template: froggeric/Qwen-Fixed-Chat-Templates