Files

T

Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)

Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.

2026-10-06 15:37:13 -07:00

3.6 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

FreeToken Benchmark Sweep — Dankserver RTX 3090

Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6. Measured 2026-09-02 with FreeToken v0.1.2 serving Qwen/Qwen3.6-35B-A3B-FP8 on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).

Methodology

Each point: restart at low KV (8k), live-rebuild upward via ft ctl cache --kv N --moe M (auto-computed MoE slot count).
Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
Measured: 2–3 iterations at 128 tokens, temperature=0, ignore_eos=true.
Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads × 256 dim × bf16). Per-slot MoE cost: ~3 MiB.
GPU budget: ~17.6 GiB after weights + fixed caches.

bs1 vs bs3 vs bs4 vs bs6

KV pool	Residency	bs1	bs3 agg	bs3/req	bs4 agg	bs4/req	bs6 agg	bs6/req
8,192	~89%	56.7	108.8	36.3	150.6	37.7	132.1	22.0
65,536	~83%	~52	106.1	35.4	148.0	37.0	131.8	22.0
131,072	~76%	~48	99.9	33.3	144.4	36.1	131.8	22.0
262,144	~62%	~48	85.3	28.4	131.3	32.8	130.5	21.8
400,000	~47%	~42	73.3	24.4	116.5	29.1	161.1	26.9
524,288	~30%	34.8	59.8	19.9	102.2	25.5	150.6	25.1
655,360	~19%	30.4	42.8	14.3	86.9	21.7	131.5	21.9
700,000	~15%	28.3	38.5	12.8	83.0	20.8	121.1	20.2

Key Findings

No cliff — gentle slope

The LRU expert cache absorbs residency reductions gracefully. Even at 5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.

Free zone up to ~262k

Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up expert residency (89% → 62%) costs nothing because the LRU working set fits in fewer slots.

bs4 is the sweet spot

bs4 dominates at every high-residency operating point. Per-request speed at bs4 (37.7 tok/s at 8k) is faster than bs3 (36.3) because 4 concurrent requests better fill the PCIe expert-streaming pipeline.

bs6 crossover at ~400k

At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the bottleneck is GPU compute on attention layers, not PCIe. At low residency (400k+), the bottleneck flips to PCIe expert streaming and bs6 wins (161 vs 116 at 400k) because more concurrency fills the pipeline.

bs3 is strictly worse than bs4

Fewer concurrent users leave PCIe idle gaps. At every point past 8k, bs3 per-request is slower than bs4 per-request. Don't run bs3 — set --max-running-requests 4 regardless of actual user count.

800k doesn't fit

4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots). The absolute ceiling is ~750k tokens. With radix prefix sharing, effective capacity exceeds naive pool ÷ concurrency.

Comparison: old llama.cpp service

The old Tiel-Coder llama.cpp service ran 200,700 total context across 3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4. FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with the caveat that it serves base Qwen3.6-35B-A3B weights, not the Tiel-Coder finetune.

3.6 KiB Raw Blame History Unescape Escape