---
name: gui-hang-remote-diagnosis
description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability"
---

# GUI Hang Remote Diagnosis

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

## Sequence

1. **Find the smoking gun** — kernel hung-task reporter already logged it:
   ```bash
   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
   ```
   Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright).
2. **Get full stacks** for victim and holder:
   ```bash
   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
   cat /proc/<holder_pid>/status | head -5   # State: R (spinning) vs D (blocked)
   cat /proc/<victim_pid>/status | head -5
   ```
   - Victim `D` + holder `R` = livelock/deadlock with lock held.
   - `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing.
3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors?
   ```bash
   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
   journalctl -b -k --no-pager | grep -i xid
   ```
   If those answer instantly with no Xid, it's a driver lock issue, not hardware.
4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events:
   ```bash
   journalctl -b --since "<start-2m>" --until "<start+1m>" --no-pager | grep -v audit
   ```
5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc.

## Recoverability verdict

- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked.
  **Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs).
- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.

## After diagnosing

- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch).
- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps.