dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Record no-spec ubatch benchmark

Browse Source
This commit is contained in:
Frank
2026-08-25 01:29:49 -07:00
parent f13124f977
commit 37fb420fcd
3 changed files with 40 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/deployment.md
+2 -1
View File
@@ -128,7 +128,8 @@ binary with expert tensors selected by an explicit `--override-tensor`
regex. Expert tensors from full-attention blocks 27 and 31 and the
output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are
CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative
decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
decoding (under re-evaluation after an 8K no-spec benchmark), f16 KV, context
491520 (3 slots of 163840 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from