command-center/agent/managed-skills/live-llm-telemetry-no-bench/SKILL.md

name, description

name	description
live-llm-telemetry-no-bench	Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU.

Live LLM telemetry without benchmarking

Use when asked "how is the model performing?" on a host that SERVES production inference, where benchmarking policy forbids running llama-bench/vllm bench beside the live service. Instead of a synthetic bench, mine the runtime's own per-request telemetry — it measures the real workload and costs nothing.

1. Where the numbers live

Runtime	Unit kind	Log command
llama.cpp (mainline, -lv 3)	root systemd unit	sudo journalctl -u <unit> --since "-3h" --no-pager
halogen-flash-server	rootless Quadlet	journalctl --user -u <unit> --no-pager (never sudo — per-user storage)

llama.cpp emits, per completed task:

I slot print_timing: id 0 | task 82 | prompt eval time =  339.51 ms /  57 tokens ( 5.96 ms per token, 167.89 tokens per second)
I slot print_timing: id 0 | task 82 |        eval time =  428.49 ms /  36 tokens (12.24 ms per token,  81.68 tokens per second)
I slot print_timing: id 0 | task 82 |       total time =  768.00 ms /  93 tokens
I slot print_timing: id 0 | task 82 |    graphs reused =         84
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 (   25 accepted /    33 generated), mean len =  3.27
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s

halogen-flash-server emits one line per request:

serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok

Grep both with grep -E "print_timing|release|serve_api: mtp". Long lines get truncated in a terminal — redirect to a file/artifact and read it whole.

2. Read the envelope, not one sample

• prompt eval on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
• eval (decode) and draft acceptance are content-dependent: measured acceptance 0.46-0.76 on one unchanged build. Never quote a single task as "the model's speed", and never compare a live sample to a deploy-time baseline without a masked llama-bench.
• graphs reused > 0 and selected slot by LCP similarity prove prefix-cache reuse is working; prompt 53 (48 cached), prefill 0.05s vs 1.58s cold is prefix caching caught in the act.
• A runtime's own window metric (tg_3s) is a cross-check on your arithmetic.

3. Attribute PCIe without touching the model

Run the counter in the background and issue ONE ordinary request:

ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
  curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
    http://127.0.0.1:7771/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
  wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'

-s t = rxpci/txpci in MB/s. Measured on a fully-VRAM-resident 27B on a 3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane link the transport is a non-factor — say so with the measured number instead of hand-waving.

4. Traps

• wait or you lose the telemetry. If the remote command exits while the curl is backgrounded, ssh closes the session and the client dies: the journal shows W srv stop: cancel task, id_task = N and NO print_timing line, while curl still reports a plausible status. Always end the compound command with wait.
• Do not run a GPU workload to "get a reading". One max_tokens=200 request is production traffic and is fine; anything that pins the card for a while is a bench and needs the stop-and-mask procedure first.
• Check which instance you are reading. Quadlet units run --replace --rm and root units get restarted: systemctl status <unit> | head -12 for the start time; task N IDs increment per request and jump across restarts.
• The engine's own ledger outranks your model of it — quote KV self size, n_slots, n_ctx_slot and the VRAM figure from the log/NVML rather than arithmetic.