0 Fork 0

Files

T

Frank 135bb2194d agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set

2026-09-17 12:27:08 -07:00

4.5 KiB

Raw Blame History

name, description

name	description
pcie-link-degradation-vs-idle-triage	Decide whether a GPU/NVMe PCIe link is genuinely degraded (bad riser, bad slot, wrong connector) or just idle-downshifted, and whether moving the card to another slot could widen it — using kernel boot bandwidth lines, port LnkCap vs LnkSta, and NVML sampled under load.

PCIe link: degraded or just idle?

Answer "is my GPU running at full speed?" without fooling yourself. The trap: an idle NVIDIA GPU reports LnkSta: Speed 2.5GT/s (downgraded) because the driver downshifts the link to Gen1 for power. That is not a fault, and it is not the slot's fault either.

Step 1 — the path ceiling, from the kernel (cheapest, most decisive)

sudo journalctl -k -b --no-pager | grep "available PCIe bandwidth"

Reads out per endpoint, e.g.:

pci 0000:c4:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
PCIe x4 link at 0000:00:03.1 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)

63.012 Gb/s = Gen4 x4 (~7.88 GB/s/direction after 128b/130b encoding).
"limited by ... link at " names the bottleneck device (usually a root port) and its width — this is the number that decides whether a swap can help.
"capable of ... x16 link" is the endpoint's own capability, not the path's. A big gap between the two is the whole story.

dmesg is often rotated; journalctl -k -b keeps this boot.

Step 2 — the two sides separately

sudo lspci -vv -s <endpoint> | grep -E "LnkCap:|LnkSta:|LnkCtl2:|LnkCap2:"
sudo lspci -vv -s <root port> | grep -E "LnkCap:|LnkSta:|LnkCtl2:"

LnkSta on a bridge is the trained link; on an endpoint it is that endpoint's view. Widths/(downgraded) compare against the own LnkCap.
LnkCtl2: Target Link Speed: 16GT/s on both ends = the platform intends Gen4, so a Gen1 current speed is power management, not misconfiguration.
A root port whose LnkCap width exceeds the trained width means the socket or adapter wires fewer lanes than the controller can drive. M.2 connectors are 4 lanes by pinout — an M.2 riser can never deliver more, regardless of what the bridge advertises.

Step 3 — prove the link under load (the step people skip)

Never conclude from an idle reading on NVIDIA. Sample NVML while the GPU works:

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,\
pcie.link.gen.hostmax,pcie.link.width.current,pcie.link.width.max --format=csv

Fire a small, legitimate request at whatever already serves on that GPU (e.g. a max_tokens completion against the local LLM endpoint) and poll the query while utilization is high:

gen.current climbing 1 → hostmax = idle downshift, link healthy.
gen.current stuck below hostmax under load = real degradation.
width.current < width.max under load = physical lane limit (wiring, riser, or connector) — no software touches this.
On a busy production GPU, this is a usage smoke test, not a benchmark: never stop/bench a live service to read its link.

Step 4 — do not plan hardware from `dmidecode -t slot`

On consumer/mini-PC firmware this table is routinely fiction — it has been seen labelling an Ethernet port and an SD-reader controller as "M.2 Socket 3" slots and advertising a free slot in a PCI domain that does not exist (e.g. 0200:00:03.2). Cross-check with the live bridge list:

lspci -nn | grep "PCI bridge"    # every root port, whether populated or empty
lspci -t                          # topology; which endpoints hang off which port

Empty root ports still enumerate, so a genuinely absent port is genuinely absent. Trust live PCI, not SMBIOS.

Step 5 — verdict on "would moving the card to another slot help?"

Compare the bottleneck device named in step 1 with the target slot's port LnkCap width. If the bottleneck is the endpoint side (riser/adapter) and the target port is wider, a move can help.
If both candidate connectors cap at the same width (two M.2 sockets = both x4), moving is a no-op — say so plainly instead of endorsing the swap.
Error counters are a useful tiebreaker: clean DevSta/CESta with a low trained speed means "wired narrow", not "failing".

Reporting

Give the ceiling in Gb/s, the card's own capability, what the idle reading meant, and whether any physical change can move the number. If the answer is "the link is already at the hardware maximum", state that and quantify whether the workload actually cares (resident weights = PCIe is nearly idle per token; CPU layer offload = it hurts).

4.5 KiB Raw Blame History