Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
47 lines
2.4 KiB
Markdown
47 lines
2.4 KiB
Markdown
---
|
||
name: gui-hang-remote-diagnosis
|
||
description: "Diagnose a frozen GUI/VT-switch failure over SSH by reading kernel hung-task evidence, decoding the lock-holder stack, and deciding recoverability"
|
||
---
|
||
|
||
# GUI Hang Remote Diagnosis
|
||
|
||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||
|
||
## Sequence
|
||
|
||
1. **Find the smoking gun** — kernel hung-task reporter already logged it:
|
||
```bash
|
||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||
```
|
||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a semaphore likely last held by task Y:PID` (modern kernels name the lock holder outright).
|
||
2. **Get full stacks** for victim and holder:
|
||
```bash
|
||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||
cat /proc/<holder_pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||
cat /proc/<victim_pid>/status | head -5
|
||
```
|
||
- Victim `D` + holder `R` = livelock/deadlock with lock held.
|
||
- `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning, not progressing.
|
||
3. **Prove whether hardware or software** — GPU/CPU still alive? No Xid errors?
|
||
```bash
|
||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||
journalctl -b -k --no-pager | grep -i xid
|
||
```
|
||
If those answer instantly with no Xid, it's a driver lock issue, not hardware.
|
||
4. **Find the trigger window**: block start = (first hung message timestamp − reported seconds). Then inspect user-session events:
|
||
```bash
|
||
journalctl -b --since "<start-2m>" --until "<start+1m>" --no-pager | grep -v audit
|
||
```
|
||
5. **Enumerate collateral**: any other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`), USB workers, etc.
|
||
|
||
## Recoverability verdict
|
||
|
||
- Compositor in `D` state → unkillable; spinning kernel thread holding the lock → cannot be poked.
|
||
**Only exit is reboot**: `sync && systemctl reboot --force --force` (double-force skips stop-jobs that wait forever on D-state PIDs).
|
||
- If holder is `D` (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.
|
||
|
||
## After diagnosing
|
||
|
||
- Map stack symbols to driver internals to name the exact subsystem (e.g. `DifrPrefetchEventDeferredWork` → NVKMS DIFR prefetch).
|
||
- Check upstream: search the exact symbol names + driver version; propose targeted off-switches (`NVreg_RegistryDwords`, module params from `/proc/driver/nvidia/params`) rather than blanket driver swaps.
|