serverAppConfigs/serverAppConfigs-docs/Operations/SSH and Remote Access.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	SSH and Remote Access	Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage.	ssh remote fish gui diagnosis	2026-10-05T00:00:00-07:00

SSH and Remote Access

Passwordless SSH bootstrap without sshpass/expect

Use when you must install your pubkey on remote hosts but sshpass and expect are not installed and you only have a password. Modern OpenSSH (≥ 8.4) can drive the password prompt from a helper program, so the password flows through the process env only — never a file, never argv.

1. Generate a key (if none):

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
   

2. Create an askpass helper that echoes the password from an env var (the helper itself stores NO secret):

   printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
   chmod 700 /tmp/askpass.sh
   

3. Copy the pubkey, forcing the askpass path and detaching from the TTY:

   env DEPLOY_PW='<password>' \
       SSH_ASKPASS=/tmp/askpass.sh \
       SSH_ASKPASS_REQUIRE=force \
       setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
   

   Password via the transient env argument only.
4. Add host aliases in ~/.ssh/config so deploy scripts resolve names (HostName, User, IdentityFile, IdentitiesOnly yes).
5. Verify with ssh -o BatchMode=yes myhost 'echo OK; hostname' — BatchMode fails fast if a password would be required; a clean success proves the key works.

Gotchas: without SSH_ASKPASS_REQUIRE=force, ssh ignores SSH_ASKPASS when a TTY is present and prompts interactively (hangs in a non-interactive tool) setsid detaches the controlling terminal — required on older OpenSSH even with force; the helper must be executable (chmod 700) and print ONLY the password; IdentitiesOnly yes stops ssh offering other agent keys that could be rejected first ("Too many authentication failures"). Passwordless sudo on the target is a separate concern (NOPASSWD / FRANK_SUDO_PW).

Remote execution on a fish login shell

Running remote commands or shipping files over SSH to a host whose login shell is fish (blankpad: user blank, reachable only as mDNS FQDN blankpad.local / 192.168.0.83 — the bare hostname does not resolve from the workstation).

1. Auth: SSHPASS='<pw>' sshpass -e ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…' — password via env, never argv (ps-visible).
2. Remote exec: ssh host 'cmd' pipes the command string to the login shell (fish). Bashisms — for … do, $?, heredoc markers — fail at parse time on fish (Missing end to balance this for loop). Always wrap compound commands: ssh host 'bash -c "for b in a b c; do command -v $b || echo MISSING; done"' (escape inner $).
3. File shipping: never combine cat > /remote/path <<EOF with a trailing bash -s in the same ssh command — cat drains stdin, so the interpreter silently runs on empty stdin and prints nothing (a silent no-op that looks like success). Ship real bytes via stdin redirect: ssh host 'cat > /remote/path' < localfile, then verify byte-truth with md5sum on both ends.
4. Exit codes: $? in an unquoted remote command is parsed by fish ($status there) — wrap in bash or drop the status echo; ssh's own exit code propagates to the caller anyway.

Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first probe died on fish loop parsing; a cat > ship silently swallowed the heredoc meant for bash -s, deploying wrong bytes until md5-compared.

Related: dankserver's login shell is ZSH — never start an unquoted word with = in an ssh one-liner (echo === aborts the whole chain).

GUI hang — remote diagnosis over SSH

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

1. Find the smoking gun — the kernel hung-task reporter already logged it:

   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
   

   Look for INFO: task X blocked for more than N seconds and blocked on a semaphore likely last held by task Y:PID (modern kernels name the lock holder outright).
2. Full stacks for victim and holder:

   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
   cat /proc/<pid>/status | head -5    # State: R (spinning) vs D (blocked)
   

   Victim D + holder R = livelock/deadlock with the lock held; voluntary_ctxt_switches in the millions on an R kthread = spinning, not progressing.
3. Hardware or software? GPU/CPU still alive, no Xid errors?

   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
   journalctl -b -k --no-pager | grep -i xid
   

   Instant answers with no Xid ⇒ driver lock issue, not hardware.
4. Trigger window: block start = first hung message timestamp − reported seconds; then inspect user-session events around it (journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit).
5. Collateral: other D tasks (ps -eo pid,stat,comm | awk '$2 ~ /D/') USB workers, etc.

Recoverability verdict: compositor in D state is unkillable and a spinning kernel thread holding the lock cannot be poked — the only exit is reboot: sync && systemctl reboot --force --force (double-force skips stop-jobs waiting forever on D-state PIDs). If the holder is D (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.

After diagnosing: map stack symbols to driver internals to name the exact subsystem, check upstream for the exact symbols + driver version, and propose targeted off-switches (NVreg_RegistryDwords, module params from /proc/driver/nvidia/params) rather than blanket driver swaps.

Related

• Fleet Ops Playbook — incident procedures that run over these sessions
• Remote Job Runner — long jobs that must survive session loss
• Deploy State Triage — the deploy flow these credentials serve