Frank
6efb0fb231
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache and paged KV with radix prefix sharing. Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size). Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used. Production config: - 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM) - 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers) - 4 max-running-requests (mamba slot pool: 24 slots) - bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1 - Prefill: ~1,130 tok/s at 4k prompt Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed). Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart. Old llama.cpp scripts archived to .AGENTTRASH/. Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in docs/LLM/freetoken-benchmark-sweep.md.
3.6 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|
| research | FreeToken Benchmark Sweep — Dankserver RTX 3090 | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
|
2026-09-02 |
FreeToken Benchmark Sweep — Dankserver RTX 3090
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
Measured 2026-09-02 with FreeToken v0.1.2 serving Qwen/Qwen3.6-35B-A3B-FP8
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
Methodology
- Each point: restart at low KV (8k), live-rebuild upward via
ft ctl cache --kv N --moe M(auto-computed MoE slot count). - Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
- Measured: 2–3 iterations at 128 tokens,
temperature=0,ignore_eos=true. - Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads × 256 dim × bf16). Per-slot MoE cost: ~3 MiB.
- GPU budget: ~17.6 GiB after weights + fixed caches.
bs1 vs bs3 vs bs4 vs bs6
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|---|---|---|---|---|---|---|---|---|
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
Key Findings
No cliff — gentle slope
The LRU expert cache absorbs residency reductions gracefully. Even at 5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
Free zone up to ~262k
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up expert residency (89% → 62%) costs nothing because the LRU working set fits in fewer slots.
bs4 is the sweet spot
bs4 dominates at every high-residency operating point. Per-request speed at bs4 (37.7 tok/s at 8k) is faster than bs3 (36.3) because 4 concurrent requests better fill the PCIe expert-streaming pipeline.
bs6 crossover at ~400k
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the bottleneck is GPU compute on attention layers, not PCIe. At low residency (400k+), the bottleneck flips to PCIe expert streaming and bs6 wins (161 vs 116 at 400k) because more concurrency fills the pipeline.
bs3 is strictly worse than bs4
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
--max-running-requests 4 regardless of actual user count.
800k doesn't fit
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
The absolute ceiling is ~750k tokens. With radix prefix sharing,
effective capacity exceeds naive pool ÷ concurrency.
Comparison: old llama.cpp service
The old Tiel-Coder llama.cpp service ran 200,700 total context across 3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4. FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with the caveat that it serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune.
See Also
- freetoken-frank-default-small — production runbook
- dealign-ornith-crack — the prior llama.cpp deployment (retired)