Record no-spec ubatch benchmark
This commit is contained in:
1 parent
f13124f977
commit
37fb420fcd
3 files changed
+40
-4
No files matched your search
@@ -75,8 +75,12 @@ On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||||
|
||||
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
|
||||
auto-detected from the target (b10121 creates the MTP draft context against
|
||||
the model file). ngram under MTP hurts on this family; MTP alone is the
|
||||
proven best. `--spec-draft-n-max 4` is the knee.
|
||||
the model file). ngram under MTP hurts on this family; MTP alone was
|
||||
previously selected for the production configuration. A later 8K API
|
||||
benchmark found that this MTP path produced lower generation throughput than
|
||||
`--spec-type none` for that workload, so speculative decoding should be
|
||||
re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the
|
||||
configured value.
|
||||
|
||||
```
|
||||
--spec-type draft-mtp
|
||||
|
||||
Reference in new issue
Block a user