Frank
219a097b39
Every payload swap a live endpoint has ever run is now one row in that endpoint's Eras table, with its rollback path in the same row. No per-attempt page, no per-swap ledger, nowhere else. - Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork on the 3090). Each carries Current shape, Eras, Laws, wire contract, files. - Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras), the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing. - Rewired: every wikilink and markdown-path reference to a folded page points at the endpoint page instead; the three Research see-also lists and the ubatch sweep's runbook link included. No anchored links existed in the set. - Regenerated both index pages as pure locators -- one row, one clause, no numbers -- so the docs_completeness locator law holds for all 153 pages. tests/docs_completeness.bats green: index links every page, no unresolved links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored, and git history holds every byte).
3.6 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|
| research | FreeToken Benchmark Sweep — Dankserver RTX 3090 | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
|
2026-09-02 |
FreeToken Benchmark Sweep — Dankserver RTX 3090
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
Measured 2026-09-02 with FreeToken v0.1.2 serving Qwen/Qwen3.6-35B-A3B-FP8
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
Methodology
- Each point: restart at low KV (8k), live-rebuild upward via
ft ctl cache --kv N --moe M(auto-computed MoE slot count). - Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
- Measured: 2–3 iterations at 128 tokens,
temperature=0,ignore_eos=true. - Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads × 256 dim × bf16). Per-slot MoE cost: ~3 MiB.
- GPU budget: ~17.6 GiB after weights + fixed caches.
bs1 vs bs3 vs bs4 vs bs6
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|---|---|---|---|---|---|---|---|---|
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
Key Findings
No cliff — gentle slope
The LRU expert cache absorbs residency reductions gracefully. Even at 5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
Free zone up to ~262k
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up expert residency (89% → 62%) costs nothing because the LRU working set fits in fewer slots.
bs4 is the sweet spot
bs4 dominates at every high-residency operating point. Per-request speed at bs4 (37.7 tok/s at 8k) is faster than bs3 (36.3) because 4 concurrent requests better fill the PCIe expert-streaming pipeline.
bs6 crossover at ~400k
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the bottleneck is GPU compute on attention layers, not PCIe. At low residency (400k+), the bottleneck flips to PCIe expert streaming and bs6 wins (161 vs 116 at 400k) because more concurrency fills the pipeline.
bs3 is strictly worse than bs4
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
--max-running-requests 4 regardless of actual user count.
800k doesn't fit
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
The absolute ceiling is ~750k tokens. With radix prefix sharing,
effective capacity exceeds naive pool ÷ concurrency.
Comparison: old llama.cpp service
The old Tiel-Coder llama.cpp service ran 200,700 total context across 3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4. FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with the caveat that it serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune.
See Also
- frank-zealot — production runbook
- dealign-ornith-crack — the prior llama.cpp deployment (retired)