Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
---
|
||
type: reference
|
||
title: FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||
description: "KV pool versus decode speed for the FP8 mixture-of-experts payload on the FreeToken runtime."
|
||
tags:
|
||
- freetoken
|
||
- benchmark
|
||
- dankserver
|
||
- frank-zealot
|
||
- qwen3.6
|
||
timestamp: 2026-09-02
|
||
---
|
||
|
||
# FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||
|
||
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
|
||
Measured 2026-09-02 with FreeToken v0.1.2 serving `Qwen/Qwen3.6-35B-A3B-FP8`
|
||
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
|
||
|
||
## Methodology
|
||
|
||
- Each point: restart at low KV (8k), live-rebuild upward via
|
||
`ft ctl cache --kv N --moe M` (auto-computed MoE slot count).
|
||
- Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
|
||
- Measured: 2–3 iterations at 128 tokens, `temperature=0`, `ignore_eos=true`.
|
||
- Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
|
||
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads ×
|
||
256 dim × bf16). Per-slot MoE cost: ~3 MiB.
|
||
- GPU budget: ~17.6 GiB after weights + fixed caches.
|
||
|
||
## bs1 vs bs3 vs bs4 vs bs6
|
||
|
||
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|
||
|---------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
|
||
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
|
||
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
|
||
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
|
||
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
|
||
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
|
||
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
|
||
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
|
||
|
||
## Key Findings
|
||
|
||
### No cliff — gentle slope
|
||
|
||
The LRU expert cache absorbs residency reductions gracefully. Even at
|
||
5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
|
||
|
||
### Free zone up to ~262k
|
||
|
||
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up
|
||
expert residency (89% → 62%) costs nothing because the LRU working set
|
||
fits in fewer slots.
|
||
|
||
### bs4 is the sweet spot
|
||
|
||
bs4 dominates at every high-residency operating point. Per-request
|
||
speed at bs4 (37.7 tok/s at 8k) is *faster* than bs3 (36.3) because 4
|
||
concurrent requests better fill the PCIe expert-streaming pipeline.
|
||
|
||
### bs6 crossover at ~400k
|
||
|
||
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the
|
||
bottleneck is GPU compute on attention layers, not PCIe. At low
|
||
residency (400k+), the bottleneck flips to PCIe expert streaming and
|
||
bs6 wins (161 vs 116 at 400k) because more concurrency fills the
|
||
pipeline.
|
||
|
||
### bs3 is strictly worse than bs4
|
||
|
||
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
|
||
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
|
||
`--max-running-requests 4` regardless of actual user count.
|
||
|
||
### 800k doesn't fit
|
||
|
||
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
|
||
The absolute ceiling is ~750k tokens. With radix prefix sharing,
|
||
effective capacity exceeds naive `pool ÷ concurrency`.
|
||
|
||
## Comparison: old llama.cpp service
|
||
|
||
The old Tiel-Coder llama.cpp service ran 200,700 total context across
|
||
3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4.
|
||
FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with
|
||
the caveat that it serves base Qwen3.6-35B-A3B weights, not the
|
||
Tiel-Coder finetune.
|
||
|
||
## See Also
|
||
|
||
- [[frank-zealot]] — production runbook
|
||
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired) |