frank-small: record 2x262144 verification numbers
Live-verified post-deploy: context_len=262144, max_running_requests=2, max_total_num_tokens=606208 in the boot log; a 200,000-token prompt (compaction height is 183,501) prefilled+completed in one request (97.3 s, ~2,056 tok/s eager); two concurrent 50k requests answered. Docs-only change.
This commit is contained in:
@@ -142,9 +142,15 @@ re-checks). The serve script refuses to start with a missing shard.
|
||||
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
|
||||
above is what makes that compile succeed).
|
||||
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
|
||||
- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
|
||||
- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of
|
||||
live KV at once).
|
||||
- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
|
||||
50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
|
||||
incl. 8 decode tokens, ~2,056 t/s).
|
||||
- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
|
||||
(216k tokens of live KV at once). 2-slot shape: two concurrent
|
||||
50,000-token requests both answered (28.8 s wall).
|
||||
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
|
||||
well past frank-primary's 183,501-token compaction trigger (70% of its
|
||||
262,144 window) — prefills and completes in ONE request.
|
||||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||||
|
||||
## Rollback
|
||||
|
||||
Reference in New Issue
Block a user