serverAppConfigs/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	FreeToken frank-default-small Runbook	FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).	freetoken dankserver frank-default-small qwen3.6 moe fp8	2026-09-02

FreeToken frank-default-small Runbook

The frank-default-small service on Dankserver is served by FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache. It replaced the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.

Model

Field	Value
HuggingFace	Qwen/Qwen3.6-35B-A3B-FP8
Architecture	qwen3_5_moe (Qwen3_5MoeForConditionalGeneration)
Quant	modelopt block-fp8 (128×128 weight_block_size)
Size on disk	35 GB
Path	/home/dank/models/qwen36-35b-a3b-fp8
Layers	40 (10 full-attention + 30 GDN/linear-attention)
Experts	256 per layer, 8 used per token
KV heads	2 × 256 dim
Context	262,144 (model max)

Important: This serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune that was previously served via llama.cpp.

Runtime

Field	Value
Engine	FreeToken v0.1.2 (ft CLI)
Venv	/home/dank/freetoken/.venv (uv-managed CPython 3.12)
Dependencies	torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi
CUDA toolchain	pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86
API	OpenAI-compatible (/v1/chat/completions, /v1/models)

Pool Configuration

The GPU budget (~17.6 GiB after weights + fixed caches) is split MoE-first between expert residency and KV pages:

Pool	Size	VRAM	Notes
MoE cache	2,048 slots (20%)	6.0 GiB	LRU expert cache, out of 10,240 total
KV cache	453,005 tokens	8.6 GiB	~20 KiB/token (10 attn layers only)
Mamba state	24 slots	1.4 GiB	Per-sequence GDN state (caps concurrency)

Live Resize

KV and MoE can be re-partitioned without restart:

ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300

• Growing KV (shrinking MoE): works live.
• Shrinking KV (growing MoE): requires restart (old + new pools cannot coexist in VRAM during the swap → OOM).
• The fit check refuses overcommit: if the plan doesn't fit, the old cache is retained and serving continues.

Benchmark Results (2026-09-02)

See freetoken-benchmark-sweep for the full bs1/bs3/bs4/bs6 sweep across KV pool sizes 8k–700k.

At the production config (20% residency, bs4):

Metric	Value
bs1 decode	~30 tok/s
bs4 aggregate	~87 tok/s
bs4 per-request	~22 tok/s
Prefill (4k)	~1,130 tok/s

Quantization Compatibility

FreeToken v0.1.2 supports these safetensor quants for qwen3_5_moe:

Format	Detection	Works
BF16 (no quant config)	default	✅ (experts too large for 60 GB RAM)
modelopt block-fp8 (128×128)	quant_method: "fp8" + weight_block_size	✅ (current)
NVFP4 (modelopt)	quant_algo: "NVFP4"	✅ (best for VRAM)
NVFP4 (compressed-tensors)	4-bit float, group_size=16, strategy=tensor_group	✅
Per-tensor FP8 (modelopt)	scalar weight_scale	✅
Per-channel FP8 (compressed-tensors)	weight_scale: [O, 1]	❌ loader bug
Quark W8A8 INT8	per-channel scales	❌ loader bug
GGUF	only gemma4 arch	❌ for qwen3_5_moe

Files

File	Purpose
scripts/serve-frank-small.sh	Serve script (launches ft serve)
scripts/frank-default-small.env	Tunables (model path, pool sizes)
scripts/frank-default-small.service	Systemd unit

Limitations

• Model identity: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
• JIT compilation: first startup compiles flashinfer/tvm-ffi kernels (~5–15 min cold). Cached afterward in ~/.cache/flashinfer and ~/.cache/tvm-ffi.
• CUDA toolchain: driver-only install (no system CUDA toolkit). The venv provides pip-installed nvcc 13.2.86 for JIT compilation. All nvidia-cu13 packages must be at the same version.
• Env var collisions: never name tunables TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads these).

Citations

• FreeToken paper: arXiv 2608.16157
• FreeToken PyPI: freetoken[accel] v0.1.2
• HuggingFace model: Qwen/Qwen3.6-35B-A3B-FP8