---
name: live-llm-telemetry-no-bench
description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
---

# Live LLM telemetry without benchmarking

Use when asked "how is the model performing?" on a host that SERVES production
inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
beside the live service. Instead of a synthetic bench, mine the runtime's own
per-request telemetry — it measures the real workload and costs nothing.

## 1. Where the numbers live

| Runtime | Unit kind | Log command |
|---|---|---|
| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |

llama.cpp emits, per completed task:

```
I slot print_timing: id 0 | task 82 | prompt eval time =  339.51 ms /  57 tokens ( 5.96 ms per token, 167.89 tokens per second)
I slot print_timing: id 0 | task 82 |        eval time =  428.49 ms /  36 tokens (12.24 ms per token,  81.68 tokens per second)
I slot print_timing: id 0 | task 82 |       total time =  768.00 ms /  93 tokens
I slot print_timing: id 0 | task 82 |    graphs reused =         84
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 (   25 accepted /    33 generated), mean len =  3.27
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
```

halogen-flash-server emits one line per request:

```
serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
```

Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
truncated in a terminal — redirect to a file/artifact and read it whole.

## 2. Read the envelope, not one sample

- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
  the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
- `eval` (decode) and `draft acceptance` are content-dependent: measured
  acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
  "the model's speed", and never compare a live sample to a deploy-time
  baseline without a masked `llama-bench`.
- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
  reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
  prefix caching caught in the act.
- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.

## 3. Attribute PCIe without touching the model

Run the counter in the background and issue ONE ordinary request:

```bash
ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
  curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
    http://127.0.0.1:7771/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
  wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
```

`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
link the transport is a non-factor — say so with the measured number instead of
hand-waving.

## 4. Traps

- **`wait` or you lose the telemetry.** If the remote command exits while the
  curl is backgrounded, ssh closes the session and the client dies: the journal
  shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
  curl still reports a plausible status. Always end the compound command with
  `wait`.
- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
  is production traffic and is fine; anything that pins the card for a while is
  a bench and needs the stop-and-mask procedure first.
- **Check which instance you are reading.** Quadlet units run `--replace --rm`
  and root units get restarted: `systemctl status <unit> | head -12` for the
  start time; `task N` IDs increment per request and jump across restarts.
- **The engine's own ledger outranks your model of it** — quote `KV self size`,
  `n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
  arithmetic.