Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
88 lines
4.5 KiB
Markdown
88 lines
4.5 KiB
Markdown
---
|
|
name: live-llm-telemetry-no-bench
|
|
description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
|
|
---
|
|
|
|
# Live LLM telemetry without benchmarking
|
|
|
|
Use when asked "how is the model performing?" on a host that SERVES production
|
|
inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
|
|
beside the live service. Instead of a synthetic bench, mine the runtime's own
|
|
per-request telemetry — it measures the real workload and costs nothing.
|
|
|
|
## 1. Where the numbers live
|
|
|
|
| Runtime | Unit kind | Log command |
|
|
|---|---|---|
|
|
| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
|
|
| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |
|
|
|
|
llama.cpp emits, per completed task:
|
|
|
|
```
|
|
I slot print_timing: id 0 | task 82 | prompt eval time = 339.51 ms / 57 tokens ( 5.96 ms per token, 167.89 tokens per second)
|
|
I slot print_timing: id 0 | task 82 | eval time = 428.49 ms / 36 tokens (12.24 ms per token, 81.68 tokens per second)
|
|
I slot print_timing: id 0 | task 82 | total time = 768.00 ms / 93 tokens
|
|
I slot print_timing: id 0 | task 82 | graphs reused = 84
|
|
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 ( 25 accepted / 33 generated), mean len = 3.27
|
|
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
|
|
```
|
|
|
|
halogen-flash-server emits one line per request:
|
|
|
|
```
|
|
serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
|
|
```
|
|
|
|
Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
|
|
truncated in a terminal — redirect to a file/artifact and read it whole.
|
|
|
|
## 2. Read the envelope, not one sample
|
|
|
|
- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
|
|
the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
|
|
- `eval` (decode) and `draft acceptance` are content-dependent: measured
|
|
acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
|
|
"the model's speed", and never compare a live sample to a deploy-time
|
|
baseline without a masked `llama-bench`.
|
|
- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
|
|
reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
|
|
prefix caching caught in the act.
|
|
- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.
|
|
|
|
## 3. Attribute PCIe without touching the model
|
|
|
|
Run the counter in the background and issue ONE ordinary request:
|
|
|
|
```bash
|
|
ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
|
|
curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
|
|
http://127.0.0.1:7771/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
|
|
wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
|
|
```
|
|
|
|
`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
|
|
3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
|
|
idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
|
|
link the transport is a non-factor — say so with the measured number instead of
|
|
hand-waving.
|
|
|
|
## 4. Traps
|
|
|
|
- **`wait` or you lose the telemetry.** If the remote command exits while the
|
|
curl is backgrounded, ssh closes the session and the client dies: the journal
|
|
shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
|
|
curl still reports a plausible status. Always end the compound command with
|
|
`wait`.
|
|
- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
|
|
is production traffic and is fine; anything that pins the card for a while is
|
|
a bench and needs the stop-and-mask procedure first.
|
|
- **Check which instance you are reading.** Quadlet units run `--replace --rm`
|
|
and root units get restarted: `systemctl status <unit> | head -12` for the
|
|
start time; `task N` IDs increment per request and jump across restarts.
|
|
- **The engine's own ledger outranks your model of it** — quote `KV self size`,
|
|
`n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
|
|
arithmetic.
|