Question it answers: within the headroom the deployment has, how large may the logical
batch and the micro-batch be before the activation buffers stop fitting — and what does
each size buy in prefill?
- One llama-bench invocation per cell. llama-bench exit(1)s the moment a test cannot run,
so a single infeasible ubatch would otherwise take every later cell with it, and
infeasibility is one of the things being looked for.
- A VRAM sampler per cell, covering the load. Batch/ubatch sizes the activation buffers,
so a cell that looks fast because its allocation is smaller than intended has to be
distinguishable from one that is genuinely fast.
- The service is stopped for the sweep and started again from a trap, because a bench
beside a live service starves the same GPU.
- Run it from a transient unit: the deploy's unit sets LimitMEMLOCK=infinity for the ROCm
staging buffers and a shell on these hosts only gets 8 MiB.
- llama-bench separates -ts/-dev with SLASHES and has no -c, --parallel or --spec-*: the
context is n_prompt + n_gen + n_depth, and the drafter cannot take part.
Frank
87845fca22