diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index c687ac5..a27d28f 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -427,6 +427,43 @@ of its own instead of queueing, at the cost of halving the window each one gets. `--parallel` splits the context and the KV between slots; it does not add throughput, so the trade is concurrency against conversation length. +### Benchmarking this endpoint + +`scripts/bench-batch-ubatch.sh` sweeps `--batch-size` against `--ubatch-size` +with `llama-bench` and records, per cell, the prefill rate, the decode rate and +each card's **peak** VRAM. Run it from a transient unit rather than a shell: + +```bash +sudo systemd-run --unit=fm-bench --collect --no-block --property=Type=oneshot \ + --property=LimitMEMLOCK=infinity --property=User=dank \ + scripts/bench-batch-ubatch.sh +``` + +Properties of `llama-bench` shape that harness, and three of them have already +produced wrong numbers on this fleet: + +- **It separates devices differently from `llama-server`.** `/` splits devices + *within one configuration*, `,` splits *sweep values* — so this service's + `--tensor-split 36,29` is `-ts 36/29` to llama-bench, and the comma form + silently benches each card on its own instead. Likewise `--device + ROCm0,ROCm1` → `-dev ROCm0/ROCm1`. +- **There is no `-c`.** The context is a consequence of the test shape: + `n_ctx = n_prompt + n_gen + n_depth`. No bench can therefore prove that a + configuration's activation buffers fit beside the *deployed* KV — that needs + the live unit at 262144 across two slots. +- **A test that cannot allocate calls `exit(1)`**, taking the whole process with + it, so the harness runs one invocation per cell and a failure costs one TSV + line instead of the rest of the sweep. +- **A shell's memlock limit is not the unit's.** The unit sets + `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell here gets + 8 MiB, which fails the load for a reason that has nothing to do with the + model being benched. +- **No `--spec-*` and no `--parallel`.** The drafter cannot take part, so a + `llama-bench` decode figure is the *drafterless* baseline — 15.9 tok/s at 64k + depth against the deployed 43.8 — and slots are never exercised. `-d ` is + what puts KV in front of the measurement at all: a `tg` instance runs with + `n_prompt = 0`, i.e. at ~zero depth unless `-d` says otherwise. + ## Observability The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace