---
type: reference
title: SSH and Remote Access
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
tags: [ssh, remote, fish, gui, diagnosis]
timestamp: 2026-10-05T00:00:00-07:00
---

# SSH and Remote Access

## Passwordless SSH bootstrap without sshpass/expect

Use when you must install your pubkey on remote hosts but `sshpass` and
`expect` are not installed and you only have a password. Modern OpenSSH
(≥ 8.4) can drive the password prompt from a helper program, so the password
flows through the **process env only** — never a file, never argv.

1. **Generate a key** (if none):
   ```bash
   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
   ```
2. **Create an askpass helper** that echoes the password from an env var (the
   helper itself stores NO secret):
   ```bash
   printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
   chmod 700 /tmp/askpass.sh
   ```
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
   ```bash
   env DEPLOY_PW='<password>' \
       SSH_ASKPASS=/tmp/askpass.sh \
       SSH_ASKPASS_REQUIRE=force \
       setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
   ```
   Password via the transient `env` argument only.
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
   (`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
   BatchMode fails fast if a password would be required; a clean success
   proves the key works.

Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
TTY is present and prompts interactively (hangs in a non-interactive tool);
`setsid` detaches the controlling terminal — required on older OpenSSH even
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
be rejected first ("Too many authentication failures"). Passwordless **sudo**
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).

## Remote execution on a fish login shell

Running remote commands or shipping files over SSH to a host whose login
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
workstation).

1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
   PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
   — password via env, never argv (ps-visible).
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
   shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
   *parse* time on fish (`Missing end to balance this for loop`). Always wrap
   compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
   echo MISSING; done"'` (escape inner `$`).
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
   trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
   interpreter silently runs on empty stdin and prints nothing (a silent
   no-op that *looks* like success). Ship real bytes via stdin redirect:
   `ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
   `md5sum` on both ends.
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
   (`$status` there) — wrap in bash or drop the status echo; ssh's own exit
   code propagates to the caller anyway.

Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
probe died on fish loop parsing; a `cat >` ship silently swallowed the
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.

Related: dankserver's login shell is ZSH — never start an unquoted word with
`=` in an ssh one-liner (`echo ===` aborts the whole chain).

## GUI hang — remote diagnosis over SSH

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

1. **Find the smoking gun** — the kernel hung-task reporter already logged
   it:
   ```bash
   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
   ```
   Look for `INFO: task X blocked for more than N seconds` and `blocked on a
   semaphore likely last held by task Y:PID` (modern kernels name the lock
   holder outright).
2. **Full stacks** for victim and holder:
   ```bash
   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
   cat /proc/<pid>/status | head -5    # State: R (spinning) vs D (blocked)
   ```
   Victim `D` + holder `R` = livelock/deadlock with the lock held;
   `voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
   not progressing.
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
   ```bash
   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
   journalctl -b -k --no-pager | grep -i xid
   ```
   Instant answers with no Xid ⇒ driver lock issue, not hardware.
4. **Trigger window**: block start = first hung message timestamp − reported
   seconds; then inspect user-session events around it
   (`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
   USB workers, etc.

**Recoverability verdict:** compositor in `D` state is unkillable and a
spinning kernel thread holding the lock cannot be poked — **the only exit is
reboot**: `sync && systemctl reboot --force --force` (double-force skips
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
something that can complete, e.g. storage), recovery may be possible by
clearing that dependency first.

After diagnosing: map stack symbols to driver internals to name the exact
subsystem, check upstream for the exact symbols + driver version, and propose
targeted off-switches (`NVreg_RegistryDwords`, module params from
`/proc/driver/nvidia/params`) rather than blanket driver swaps.

## Related

- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
- [[Remote Job Runner]] — long jobs that must survive session loss
- [[Deploy State Triage]] — the deploy flow these credentials serve