Files

T

Frank 6efb0fb231 feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8

Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with
FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert
weights in host RAM with an LRU GPU cache and paged KV with radix prefix
sharing.

Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size).
Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used.

Production config:
- 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM)
- 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers)
- 4 max-running-requests (mamba slot pool: 24 slots)
- bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1
- Prefill: ~1,130 tok/s at 4k prompt

Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor
FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed).
Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart.

Old llama.cpp scripts archived to .AGENTTRASH/.
Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in
docs/LLM/freetoken-benchmark-sweep.md.

2026-08-27 02:01:16 -07:00

4.4 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

FreeToken frank-default-small Runbook

The frank-default-small service on Dankserver is served by FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache. It replaced the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.

Model

Field	Value
HuggingFace	`Qwen/Qwen3.6-35B-A3B-FP8`
Architecture	`qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration)
Quant	modelopt block-fp8 (128×128 `weight_block_size`)
Size on disk	35 GB
Path	`/home/dank/models/qwen36-35b-a3b-fp8`
Layers	40 (10 full-attention + 30 GDN/linear-attention)
Experts	256 per layer, 8 used per token
KV heads	2 × 256 dim
Context	262,144 (model max)

Important: This serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune that was previously served via llama.cpp.

Runtime

Field	Value
Engine	FreeToken v0.1.2 (`ft` CLI)
Venv	`/home/dank/freetoken/.venv` (uv-managed CPython 3.12)
Dependencies	torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi
CUDA toolchain	pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86
API	OpenAI-compatible (`/v1/chat/completions`, `/v1/models`)

Pool Configuration

The GPU budget (~17.6 GiB after weights + fixed caches) is split MoE-first between expert residency and KV pages:

Pool	Size	VRAM	Notes
MoE cache	2,048 slots (20%)	6.0 GiB	LRU expert cache, out of 10,240 total
KV cache	453,005 tokens	8.6 GiB	~20 KiB/token (10 attn layers only)
Mamba state	24 slots	1.4 GiB	Per-sequence GDN state (caps concurrency)

Live Resize

KV and MoE can be re-partitioned without restart:

ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300

Growing KV (shrinking MoE): works live.
Shrinking KV (growing MoE): requires restart (old + new pools cannot coexist in VRAM during the swap → OOM).
The fit check refuses overcommit: if the plan doesn't fit, the old cache is retained and serving continues.

Benchmark Results (2026-09-02)

See freetoken-benchmark-sweep for the full bs1/bs3/bs4/bs6 sweep across KV pool sizes 8k–700k.

At the production config (20% residency, bs4):

Metric	Value
bs1 decode	~30 tok/s
bs4 aggregate	~87 tok/s
bs4 per-request	~22 tok/s
Prefill (4k)	~1,130 tok/s

Quantization Compatibility

FreeToken v0.1.2 supports these safetensor quants for qwen3_5_moe:

Format	Detection	Works
BF16 (no quant config)	default	✅ (experts too large for 60 GB RAM)
modelopt block-fp8 (128×128)	`quant_method: "fp8"` + `weight_block_size`	✅ (current)
NVFP4 (modelopt)	`quant_algo: "NVFP4"`	✅ (best for VRAM)
NVFP4 (compressed-tensors)	4-bit float, group_size=16, strategy=tensor_group	✅
Per-tensor FP8 (modelopt)	scalar `weight_scale`	✅
Per-channel FP8 (compressed-tensors)	`weight_scale: [O, 1]`	❌ loader bug
Quark W8A8 INT8	per-channel scales	❌ loader bug
GGUF	only `gemma4` arch	❌ for qwen3_5_moe

Files

File	Purpose
`scripts/serve-frank-small.sh`	Serve script (launches `ft serve`)
`scripts/frank-default-small.env`	Tunables (model path, pool sizes)
`scripts/frank-default-small.service`	Systemd unit

Limitations

Model identity: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
JIT compilation: first startup compiles flashinfer/tvm-ffi kernels (~5–15 min cold). Cached afterward in ~/.cache/flashinfer and ~/.cache/tvm-ffi.
CUDA toolchain: driver-only install (no system CUDA toolkit). The venv provides pip-installed nvcc 13.2.86 for JIT compilation. All nvidia-cu13 packages must be at the same version.
Env var collisions: never name tunables TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads these).

Citations

FreeToken paper: arXiv 2608.16157
FreeToken PyPI: freetoken[accel] v0.1.2
HuggingFace model: Qwen/Qwen3.6-35B-A3B-FP8

4.4 KiB Raw Permalink Blame History Unescape Escape