Record no-spec ubatch benchmark

docs/LLM/dealign-ornith-crack.md

@@ -75,8 +75,12 @@ On Dankserver the available buffer types are `CPU` and `CUDA0`.
 
 MoE model with a bundled MTP head -> native `draft-mtp`. The head is
 auto-detected from the target (b10121 creates the MTP draft context against
-the model file). ngram under MTP hurts on this family; MTP alone is the
-proven best. `--spec-draft-n-max 4` is the knee.
+the model file). ngram under MTP hurts on this family; MTP alone was
+previously selected for the production configuration. A later 8K API
+benchmark found that this MTP path produced lower generation throughput than
+`--spec-type none` for that workload, so speculative decoding should be
+re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the
+configured value.
 
 ```
 --spec-type draft-mtp

docs/LLM/deployment.md

@@ -128,7 +128,8 @@ binary with expert tensors selected by an explicit `--override-tensor`
 regex. Expert tensors from full-attention blocks 27 and 31 and the
 output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are
 CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative
-decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
+decoding (under re-evaluation after an 8K no-spec benchmark), f16 KV, context
+491520 (3 slots of 163840 each), batch
 4096, ubatch 2048, and `--no-mmap`. The unit is
 `scripts/frank-default-small.service`. It launches through
 `scripts/serve-frank-small.sh`, which reads tunables from

docs/LLM/frank-default-small-benchmark.md

@@ -136,12 +136,43 @@ context pool.
 If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
 (generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
 
+## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
+
+Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
+and hybrid expert placement, but explicitly disabled speculative decoding with
+`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
+`--batch-size 4096`, prompt caching disabled, and one unique approximately
+8K-token prompt per point. The production service was stopped during the
+sweep and automatically restarted afterward.
+
+| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
+|---:|---:|---:|---:|
+| 1024 | 1944 | 53.91 | 248 MiB |
+| 1280 | 2030 | 53.87 | 248 MiB |
+| 1536 | 2190 | 53.66 | 248 MiB |
+| 1792 | 2203 | 53.69 | 248 MiB |
+| 2048 | 2423 | 53.80 | 248 MiB |
+
+Prefill improves through `ubatch=2048`, while generation is effectively flat.
+The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
+larger value was not tested in this corrected no-spec run.
+
+The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
+generation tokens/s at these `ubatch` values, with variable draft acceptance.
+The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
+means MTP was a net throughput loss for this workload and configuration,
+despite accepting draft tokens. The production service remains on MTP until a
+separate decision is made; acceptance count alone is not sufficient to select
+speculation.
+
 ## Limitations
 
 llama-bench measures raw decode and prefill. It does not run speculative
 decoding, parallel slots, or the mmproj vision tower. The live service adds
 those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
-real overhead varies a little with load.
+real overhead varies a little with load. The Ornith sweep above used the
+OpenAI-compatible server API rather than `llama-bench`, and its throughput
+includes the server's timing instrumentation.
 
 ## See Also