agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: live-llm-telemetry-no-bench
|
||||
description: "Measure a LIVE LLM endpoint's real prefill/decode/spec-decode performance and its PCIe cost from the runtime's own journald telemetry plus nvidia-smi dmon, without stopping the service or running a benchmark beside it — for hosts whose rules forbid benchmarking a serving GPU."
|
||||
---
|
||||
|
||||
# Live LLM telemetry without benchmarking
|
||||
|
||||
Use when asked "how is the model performing?" on a host that SERVES production
|
||||
inference, where benchmarking policy forbids running `llama-bench`/`vllm bench`
|
||||
beside the live service. Instead of a synthetic bench, mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
## 1. Where the numbers live
|
||||
|
||||
| Runtime | Unit kind | Log command |
|
||||
|---|---|---|
|
||||
| llama.cpp (mainline, `-lv 3`) | root systemd unit | `sudo journalctl -u <unit> --since "-3h" --no-pager` |
|
||||
| halogen-flash-server | rootless Quadlet | `journalctl --user -u <unit> --no-pager` (**never** sudo — per-user storage) |
|
||||
|
||||
llama.cpp emits, per completed task:
|
||||
|
||||
```
|
||||
I slot print_timing: id 0 | task 82 | prompt eval time = 339.51 ms / 57 tokens ( 5.96 ms per token, 167.89 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | eval time = 428.49 ms / 36 tokens (12.24 ms per token, 81.68 tokens per second)
|
||||
I slot print_timing: id 0 | task 82 | total time = 768.00 ms / 93 tokens
|
||||
I slot print_timing: id 0 | task 82 | graphs reused = 84
|
||||
I slot print_timing: id 0 | task 82 | draft acceptance = 0.75758 ( 25 accepted / 33 generated), mean len = 3.27
|
||||
I slot print_timing: id 0 | task 97 | n_gen = 188, tg = 62.06 t/s, tg_3s = 62.39 t/s
|
||||
```
|
||||
|
||||
halogen-flash-server emits one line per request:
|
||||
|
||||
```
|
||||
serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16 rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok
|
||||
```
|
||||
|
||||
Grep both with `grep -E "print_timing|release|serve_api: mtp"`. Long lines get
|
||||
truncated in a terminal — redirect to a file/artifact and read it whole.
|
||||
|
||||
## 2. Read the envelope, not one sample
|
||||
|
||||
- `prompt eval` on a SHORT prompt is overhead-dominated (57 tok @ 168 t/s) — use
|
||||
the longest prompt available for the prefill figure (1106 tok @ 1013 t/s).
|
||||
- `eval` (decode) and `draft acceptance` are content-dependent: measured
|
||||
acceptance 0.46-0.76 on one unchanged build. Never quote a single task as
|
||||
"the model's speed", and never compare a live sample to a deploy-time
|
||||
baseline without a masked `llama-bench`.
|
||||
- `graphs reused` > 0 and `selected slot by LCP similarity` prove prefix-cache
|
||||
reuse is working; `prompt 53 (48 cached), prefill 0.05s` vs `1.58s` cold is
|
||||
prefix caching caught in the act.
|
||||
- A runtime's own window metric (`tg_3s`) is a cross-check on your arithmetic.
|
||||
|
||||
## 3. Attribute PCIe without touching the model
|
||||
|
||||
Run the counter in the background and issue ONE ordinary request:
|
||||
|
||||
```bash
|
||||
ssh host 'nvidia-smi dmon -s t -c 16 -o T & sleep 0.5; \
|
||||
curl -s -m 60 -o /dev/null -w "http=%{http_code} wall=%{time_total}s\n" \
|
||||
http://127.0.0.1:7771/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"model\":\"<alias>\",\"messages\":[{\"role\":\"user\",\"content\":\"...\"}],\"max_tokens\":200}"; \
|
||||
wait; sudo journalctl -u <unit> --since "-1min" --no-pager | grep print_timing'
|
||||
```
|
||||
|
||||
`-s t` = `rxpci`/`txpci` in MB/s. Measured on a fully-VRAM-resident 27B on a
|
||||
3090: peak ~230 MB/s against a Gen4 x4 link worth ~7,900 MB/s = ~3%, and 0 at
|
||||
idle. Decode traffic is essentially the sampled-logits stream, so on a 4-lane
|
||||
link the transport is a non-factor — say so with the measured number instead of
|
||||
hand-waving.
|
||||
|
||||
## 4. Traps
|
||||
|
||||
- **`wait` or you lose the telemetry.** If the remote command exits while the
|
||||
curl is backgrounded, ssh closes the session and the client dies: the journal
|
||||
shows `W srv stop: cancel task, id_task = N` and NO `print_timing` line, while
|
||||
curl still reports a plausible status. Always end the compound command with
|
||||
`wait`.
|
||||
- **Do not run a GPU workload to "get a reading".** One max_tokens=200 request
|
||||
is production traffic and is fine; anything that pins the card for a while is
|
||||
a bench and needs the stop-and-mask procedure first.
|
||||
- **Check which instance you are reading.** Quadlet units run `--replace --rm`
|
||||
and root units get restarted: `systemctl status <unit> | head -12` for the
|
||||
start time; `task N` IDs increment per request and jump across restarts.
|
||||
- **The engine's own ledger outranks your model of it** — quote `KV self size`,
|
||||
`n_slots`, `n_ctx_slot` and the VRAM figure from the log/NVML rather than
|
||||
arithmetic.
|
||||
Reference in new issue
Block a user