Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
102 lines
4.5 KiB
Markdown
102 lines
4.5 KiB
Markdown
---
|
|
name: pcie-link-degradation-vs-idle-triage
|
|
description: "Decide whether a GPU/NVMe PCIe link is genuinely degraded (bad riser, bad slot, wrong connector) or just idle-downshifted, and whether moving the card to another slot could widen it — using kernel boot bandwidth lines, port LnkCap vs LnkSta, and NVML sampled under load."
|
|
---
|
|
|
|
# PCIe link: degraded or just idle?
|
|
|
|
Answer "is my GPU running at full speed?" without fooling yourself. The trap: an
|
|
idle NVIDIA GPU reports `LnkSta: Speed 2.5GT/s (downgraded)` because the driver
|
|
downshifts the link to Gen1 for power. That is not a fault, and it is not the
|
|
slot's fault either.
|
|
|
|
## Step 1 — the path ceiling, from the kernel (cheapest, most decisive)
|
|
|
|
```bash
|
|
sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"
|
|
```
|
|
|
|
Reads out per endpoint, e.g.:
|
|
|
|
```
|
|
pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
|
|
PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
|
|
```
|
|
|
|
- `63.012 Gb/s` = Gen4 **x4** (~7.88 GB/s/direction after 128b/130b encoding).
|
|
- "limited by ... link at <BDF>" names the **bottleneck device** (usually a root
|
|
port) and its width — this is the number that decides whether a swap can help.
|
|
- "capable of ... x16 link" is the **endpoint's own** capability, not the path's.
|
|
A big gap between the two is the whole story.
|
|
|
|
`dmesg` is often rotated; `journalctl -k -b` keeps this boot.
|
|
|
|
## Step 2 — the two sides separately
|
|
|
|
```bash
|
|
sudo lspci -vv -s <endpoint> | grep -E "LnkCap:|LnkSta:|LnkCtl2:|LnkCap2:"
|
|
sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkSta:|LnkCtl2:"
|
|
```
|
|
|
|
- `LnkSta` on a *bridge* is the trained link; on an endpoint it is that
|
|
endpoint's view. Widths/`(downgraded)` compare against the **own** `LnkCap`.
|
|
- `LnkCtl2: Target Link Speed: 16GT/s` on both ends = the platform *intends*
|
|
Gen4, so a Gen1 current speed is power management, not misconfiguration.
|
|
- A root port whose `LnkCap` width exceeds the trained width means the socket or
|
|
adapter wires fewer lanes than the controller can drive. **M.2 connectors are
|
|
4 lanes by pinout** — an M.2 riser can never deliver more, regardless of what
|
|
the bridge advertises.
|
|
|
|
## Step 3 — prove the link under load (the step people skip)
|
|
|
|
Never conclude from an idle reading on NVIDIA. Sample NVML while the GPU works:
|
|
|
|
```bash
|
|
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,\
|
|
pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv
|
|
```
|
|
|
|
Fire a small, legitimate request at whatever already serves on that GPU (e.g. a
|
|
`max_tokens` completion against the local LLM endpoint) and poll the query while
|
|
utilization is high:
|
|
|
|
- `gen.current` climbing 1 → `hostmax` = idle downshift, link healthy.
|
|
- `gen.current` stuck below `hostmax` under load = real degradation.
|
|
- `width.current < width.max` under load = **physical lane limit** (wiring,
|
|
riser, or connector) — no software touches this.
|
|
- On a busy production GPU, this is a usage smoke test, not a benchmark: never
|
|
stop/bench a live service to read its link.
|
|
|
|
## Step 4 — do not plan hardware from `dmidecode -t slot`
|
|
|
|
On consumer/mini-PC firmware this table is routinely fiction — it has been seen
|
|
labelling an Ethernet port and an SD-reader controller as "M.2 Socket 3" slots
|
|
and advertising a free slot in a **PCI domain that does not exist** (e.g.
|
|
`0200:00:03.2`). Cross-check with the live bridge list:
|
|
|
|
```bash
|
|
lspci -nn | grep "PCI bridge" # every root port, whether populated or empty
|
|
lspci -t # topology; which endpoints hang off which port
|
|
```
|
|
|
|
Empty root ports still enumerate, so a genuinely absent port is genuinely
|
|
absent. Trust live PCI, not SMBIOS.
|
|
|
|
## Step 5 — verdict on "would moving the card to another slot help?"
|
|
|
|
- Compare the **bottleneck device named in step 1** with the target slot's port
|
|
`LnkCap` width. If the bottleneck is the *endpoint side* (riser/adapter) and
|
|
the target port is wider, a move can help.
|
|
- If both candidate connectors cap at the same width (two M.2 sockets = both
|
|
x4), moving is a **no-op** — say so plainly instead of endorsing the swap.
|
|
- Error counters are a useful tiebreaker: clean `DevSta`/`CESta` with a low
|
|
trained speed means "wired narrow", not "failing".
|
|
|
|
## Reporting
|
|
|
|
Give the ceiling in Gb/s, the card's own capability, what the idle reading
|
|
meant, and whether any physical change can move the number. If the answer is
|
|
"the link is already at the hardware maximum", state that and quantify whether
|
|
the workload actually cares (resident weights = PCIe is nearly idle per token;
|
|
CPU layer offload = it hurts).
|