command-center/agent/managed-skills/gui-hang-remote-diagnosis/SKILL.md

name, description

name	description
gui-hang-remote-diagnosis	Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability

GUI Hang Remote Diagnosis

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

Sequence

1. Find the smoking gun — kernel hung-task reporter already logged it:

   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
   

   Look for INFO: task X blocked for more than N seconds and blocked on a semaphore likely last held by task Y:PID (modern kernels name the lock holder outright).
2. Get full stacks for victim and holder:

   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
   cat /proc/<holder_pid>/status | head -5   # State: R (spinning) vs D (blocked)
   cat /proc/<victim_pid>/status | head -5
   

   • Victim D + holder R = livelock/deadlock with lock held.
   • voluntary_ctxt_switches in the millions on an R kthread = spinning, not progressing.
3. Prove whether hardware or software — GPU/CPU still alive? No Xid errors?

   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
   journalctl -b -k --no-pager | grep -i xid
   

   If those answer instantly with no Xid, it's a driver lock issue, not hardware.
4. Find the trigger window: block start = (first hung message timestamp − reported seconds). Then inspect user-session events:

   journalctl -b --since "<start-2m>" --until "<start+1m>" --no-pager | grep -v audit
   

5. Enumerate collateral: any other D tasks (ps -eo pid,stat,comm | awk '$2 ~ /D/'), USB workers, etc.

Recoverability verdict

• Compositor in D state → unkillable; spinning kernel thread holding the lock → cannot be poked. Only exit is reboot: sync && systemctl reboot --force --force (double-force skips stop-jobs that wait forever on D-state PIDs).
• If holder is D (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.

After diagnosing

• Map stack symbols to driver internals to name the exact subsystem (e.g. DifrPrefetchEventDeferredWork → NVKMS DIFR prefetch).
• Check upstream: search the exact symbol names + driver version; propose targeted off-switches (NVreg_RegistryDwords, module params from /proc/driver/nvidia/params) rather than blanket driver swaps.