feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache and paged KV with radix prefix sharing. Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size). Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used. Production config: - 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM) - 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers) - 4 max-running-requests (mamba slot pool: 24 slots) - bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1 - Prefill: ~1,130 tok/s at 4k prompt Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed). Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart. Old llama.cpp scripts archived to .AGENTTRASH/. Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in docs/LLM/freetoken-benchmark-sweep.md.
This commit is contained in:
Frank committed
1 parent
e219fdf939
commit
6efb0fb231
14 files changed
+494
-470
No files matched your search
@@ -0,0 +1,127 @@
|
||||
---
|
||||
type: configuration
|
||||
title: FreeToken frank-default-small Runbook
|
||||
description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).
|
||||
tags:
|
||||
- freetoken
|
||||
- dankserver
|
||||
- frank-default-small
|
||||
- qwen3.6
|
||||
- moe
|
||||
- fp8
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
# FreeToken frank-default-small Runbook
|
||||
|
||||
The `frank-default-small` service on Dankserver is served by
|
||||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||||
expert weights in host RAM with an LRU GPU cache. It replaced the
|
||||
llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.
|
||||
|
||||
## Model
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` |
|
||||
| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
|
||||
| Quant | modelopt block-fp8 (128×128 `weight_block_size`) |
|
||||
| Size on disk | 35 GB |
|
||||
| Path | `/home/dank/models/qwen36-35b-a3b-fp8` |
|
||||
| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
|
||||
| Experts | 256 per layer, 8 used per token |
|
||||
| KV heads | 2 × 256 dim |
|
||||
| Context | 262,144 (model max) |
|
||||
|
||||
**Important:** This serves base Qwen3.6-35B-A3B weights, not the
|
||||
Tiel-Coder finetune that was previously served via llama.cpp.
|
||||
|
||||
## Runtime
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Engine | FreeToken v0.1.2 (`ft` CLI) |
|
||||
| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
|
||||
| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
|
||||
| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
|
||||
| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
|
||||
|
||||
## Pool Configuration
|
||||
|
||||
The GPU budget (~17.6 GiB after weights + fixed caches) is split
|
||||
MoE-first between expert residency and KV pages:
|
||||
|
||||
| Pool | Size | VRAM | Notes |
|
||||
|------|------|------|-------|
|
||||
| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
|
||||
| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
|
||||
| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
|
||||
|
||||
### Live Resize
|
||||
|
||||
KV and MoE can be re-partitioned without restart:
|
||||
|
||||
```bash
|
||||
ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
|
||||
```
|
||||
|
||||
- **Growing KV** (shrinking MoE): works live.
|
||||
- **Shrinking KV** (growing MoE): requires restart (old + new pools
|
||||
cannot coexist in VRAM during the swap → OOM).
|
||||
- The fit check refuses overcommit: if the plan doesn't fit, the old
|
||||
cache is retained and serving continues.
|
||||
|
||||
### Benchmark Results (2026-09-02)
|
||||
|
||||
See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
|
||||
across KV pool sizes 8k–700k.
|
||||
|
||||
At the production config (20% residency, bs4):
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| bs1 decode | ~30 tok/s |
|
||||
| bs4 aggregate | ~87 tok/s |
|
||||
| bs4 per-request | ~22 tok/s |
|
||||
| Prefill (4k) | ~1,130 tok/s |
|
||||
|
||||
## Quantization Compatibility
|
||||
|
||||
FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
|
||||
|
||||
| Format | Detection | Works |
|
||||
|--------|-----------|-------|
|
||||
| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
|
||||
| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
|
||||
| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
|
||||
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
|
||||
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
|
||||
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
|
||||
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
|
||||
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-default-small.service` | Systemd unit |
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
|
||||
- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
|
||||
(~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
|
||||
`~/.cache/tvm-ffi`.
|
||||
- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
|
||||
venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
|
||||
nvidia-cu13 packages must be at the same version.
|
||||
- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
|
||||
`GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
|
||||
|
||||
## Citations
|
||||
|
||||
- FreeToken paper: arXiv 2608.16157
|
||||
- FreeToken PyPI: `freetoken[accel]` v0.1.2
|
||||
- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8`
|
||||
Reference in new issue
Block a user