Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
3.6 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|
| reference | FreeToken Benchmark Sweep — Dankserver RTX 3090 | KV pool versus decode speed for the FP8 mixture-of-experts payload on the FreeToken runtime. |
|
2026-09-02 |
FreeToken Benchmark Sweep — Dankserver RTX 3090
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
Measured 2026-09-02 with FreeToken v0.1.2 serving Qwen/Qwen3.6-35B-A3B-FP8
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
Methodology
- Each point: restart at low KV (8k), live-rebuild upward via
ft ctl cache --kv N --moe M(auto-computed MoE slot count). - Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
- Measured: 2–3 iterations at 128 tokens,
temperature=0,ignore_eos=true. - Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads × 256 dim × bf16). Per-slot MoE cost: ~3 MiB.
- GPU budget: ~17.6 GiB after weights + fixed caches.
bs1 vs bs3 vs bs4 vs bs6
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|---|---|---|---|---|---|---|---|---|
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
Key Findings
No cliff — gentle slope
The LRU expert cache absorbs residency reductions gracefully. Even at 5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
Free zone up to ~262k
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up expert residency (89% → 62%) costs nothing because the LRU working set fits in fewer slots.
bs4 is the sweet spot
bs4 dominates at every high-residency operating point. Per-request speed at bs4 (37.7 tok/s at 8k) is faster than bs3 (36.3) because 4 concurrent requests better fill the PCIe expert-streaming pipeline.
bs6 crossover at ~400k
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the bottleneck is GPU compute on attention layers, not PCIe. At low residency (400k+), the bottleneck flips to PCIe expert streaming and bs6 wins (161 vs 116 at 400k) because more concurrency fills the pipeline.
bs3 is strictly worse than bs4
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
--max-running-requests 4 regardless of actual user count.
800k doesn't fit
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
The absolute ceiling is ~750k tokens. With radix prefix sharing,
effective capacity exceeds naive pool ÷ concurrency.
Comparison: old llama.cpp service
The old Tiel-Coder llama.cpp service ran 200,700 total context across 3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4. FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with the caveat that it serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune.
See Also
- frank-zealot — production runbook
- dealign-ornith-crack — the prior llama.cpp deployment (retired)