feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache and paged KV with radix prefix sharing. Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size). Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used. Production config: - 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM) - 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers) - 4 max-running-requests (mamba slot pool: 24 slots) - bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1 - Prefill: ~1,130 tok/s at 4k prompt Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed). Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart. Old llama.cpp scripts archived to .AGENTTRASH/. Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in docs/LLM/freetoken-benchmark-sweep.md.
This commit is contained in:
Frank
@@ -0,0 +1,94 @@
|
||||
---
|
||||
type: research
|
||||
title: FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||||
description: KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2.
|
||||
tags:
|
||||
- freetoken
|
||||
- benchmark
|
||||
- dankserver
|
||||
- frank-default-small
|
||||
- qwen3.6
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
# FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||||
|
||||
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
|
||||
Measured 2026-09-02 with FreeToken v0.1.2 serving `Qwen/Qwen3.6-35B-A3B-FP8`
|
||||
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
|
||||
|
||||
## Methodology
|
||||
|
||||
- Each point: restart at low KV (8k), live-rebuild upward via
|
||||
`ft ctl cache --kv N --moe M` (auto-computed MoE slot count).
|
||||
- Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
|
||||
- Measured: 2–3 iterations at 128 tokens, `temperature=0`, `ignore_eos=true`.
|
||||
- Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
|
||||
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads ×
|
||||
256 dim × bf16). Per-slot MoE cost: ~3 MiB.
|
||||
- GPU budget: ~17.6 GiB after weights + fixed caches.
|
||||
|
||||
## bs1 vs bs3 vs bs4 vs bs6
|
||||
|
||||
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|
||||
|---------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
|
||||
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
|
||||
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
|
||||
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
|
||||
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
|
||||
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
|
||||
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
|
||||
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
|
||||
|
||||
## Key Findings
|
||||
|
||||
### No cliff — gentle slope
|
||||
|
||||
The LRU expert cache absorbs residency reductions gracefully. Even at
|
||||
5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
|
||||
|
||||
### Free zone up to ~262k
|
||||
|
||||
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up
|
||||
expert residency (89% → 62%) costs nothing because the LRU working set
|
||||
fits in fewer slots.
|
||||
|
||||
### bs4 is the sweet spot
|
||||
|
||||
bs4 dominates at every high-residency operating point. Per-request
|
||||
speed at bs4 (37.7 tok/s at 8k) is *faster* than bs3 (36.3) because 4
|
||||
concurrent requests better fill the PCIe expert-streaming pipeline.
|
||||
|
||||
### bs6 crossover at ~400k
|
||||
|
||||
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the
|
||||
bottleneck is GPU compute on attention layers, not PCIe. At low
|
||||
residency (400k+), the bottleneck flips to PCIe expert streaming and
|
||||
bs6 wins (161 vs 116 at 400k) because more concurrency fills the
|
||||
pipeline.
|
||||
|
||||
### bs3 is strictly worse than bs4
|
||||
|
||||
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
|
||||
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
|
||||
`--max-running-requests 4` regardless of actual user count.
|
||||
|
||||
### 800k doesn't fit
|
||||
|
||||
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
|
||||
The absolute ceiling is ~750k tokens. With radix prefix sharing,
|
||||
effective capacity exceeds naive `pool ÷ concurrency`.
|
||||
|
||||
## Comparison: old llama.cpp service
|
||||
|
||||
The old Tiel-Coder llama.cpp service ran 200,700 total context across
|
||||
3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4.
|
||||
FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with
|
||||
the caveat that it serves base Qwen3.6-35B-A3B weights, not the
|
||||
Tiel-Coder finetune.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[freetoken-frank-default-small]] — production runbook
|
||||
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
type: configuration
|
||||
title: FreeToken frank-default-small Runbook
|
||||
description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).
|
||||
tags:
|
||||
- freetoken
|
||||
- dankserver
|
||||
- frank-default-small
|
||||
- qwen3.6
|
||||
- moe
|
||||
- fp8
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
# FreeToken frank-default-small Runbook
|
||||
|
||||
The `frank-default-small` service on Dankserver is served by
|
||||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||||
expert weights in host RAM with an LRU GPU cache. It replaced the
|
||||
llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.
|
||||
|
||||
## Model
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` |
|
||||
| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
|
||||
| Quant | modelopt block-fp8 (128×128 `weight_block_size`) |
|
||||
| Size on disk | 35 GB |
|
||||
| Path | `/home/dank/models/qwen36-35b-a3b-fp8` |
|
||||
| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
|
||||
| Experts | 256 per layer, 8 used per token |
|
||||
| KV heads | 2 × 256 dim |
|
||||
| Context | 262,144 (model max) |
|
||||
|
||||
**Important:** This serves base Qwen3.6-35B-A3B weights, not the
|
||||
Tiel-Coder finetune that was previously served via llama.cpp.
|
||||
|
||||
## Runtime
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Engine | FreeToken v0.1.2 (`ft` CLI) |
|
||||
| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
|
||||
| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
|
||||
| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
|
||||
| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
|
||||
|
||||
## Pool Configuration
|
||||
|
||||
The GPU budget (~17.6 GiB after weights + fixed caches) is split
|
||||
MoE-first between expert residency and KV pages:
|
||||
|
||||
| Pool | Size | VRAM | Notes |
|
||||
|------|------|------|-------|
|
||||
| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
|
||||
| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
|
||||
| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
|
||||
|
||||
### Live Resize
|
||||
|
||||
KV and MoE can be re-partitioned without restart:
|
||||
|
||||
```bash
|
||||
ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
|
||||
```
|
||||
|
||||
- **Growing KV** (shrinking MoE): works live.
|
||||
- **Shrinking KV** (growing MoE): requires restart (old + new pools
|
||||
cannot coexist in VRAM during the swap → OOM).
|
||||
- The fit check refuses overcommit: if the plan doesn't fit, the old
|
||||
cache is retained and serving continues.
|
||||
|
||||
### Benchmark Results (2026-09-02)
|
||||
|
||||
See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
|
||||
across KV pool sizes 8k–700k.
|
||||
|
||||
At the production config (20% residency, bs4):
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| bs1 decode | ~30 tok/s |
|
||||
| bs4 aggregate | ~87 tok/s |
|
||||
| bs4 per-request | ~22 tok/s |
|
||||
| Prefill (4k) | ~1,130 tok/s |
|
||||
|
||||
## Quantization Compatibility
|
||||
|
||||
FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
|
||||
|
||||
| Format | Detection | Works |
|
||||
|--------|-----------|-------|
|
||||
| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
|
||||
| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
|
||||
| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
|
||||
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
|
||||
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
|
||||
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
|
||||
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
|
||||
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-default-small.service` | Systemd unit |
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
|
||||
- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
|
||||
(~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
|
||||
`~/.cache/tvm-ffi`.
|
||||
- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
|
||||
venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
|
||||
nvidia-cu13 packages must be at the same version.
|
||||
- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
|
||||
`GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
|
||||
|
||||
## Citations
|
||||
|
||||
- FreeToken paper: arXiv 2608.16157
|
||||
- FreeToken PyPI: `freetoken[accel]` v0.1.2
|
||||
- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8`
|
||||
+14
-9
@@ -1,8 +1,8 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
|
||||
@@ -16,11 +16,11 @@ services run the Qwen-Sharp chat template.
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B2[ik_llama.cpp build b166e269
|
||||
A[RTX 3090 24GB + 60GB RAM]
|
||||
B2[FreeToken v0.1.2
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
no speculation, q8_0 + Hadamard cache, 491520 ctx]
|
||||
C[Qwen3.6-35B-A3B-FP8
|
||||
block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
@@ -46,6 +46,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[freetoken-frank-default-small]] — the active frank-default-small
|
||||
runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver
|
||||
(2026-09-02)
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
|
||||
@@ -54,9 +59,9 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
|
||||
+4
-2
@@ -105,13 +105,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for frank-default-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box hardware
|
||||
|
||||
Reference in New Issue
Block a user