serverAppConfigs/serverAppConfigs-docs/Operations/SSH and Remote Access.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	SSH and Remote Access	Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.	ssh remote fish gui diagnosis	2026-10-05T00:00:00-07:00

SSH and Remote Access

Passwordless SSH bootstrap without sshpass/expect

Use when you must install your pubkey on remote hosts but sshpass and expect are not installed and you only have a password. Modern OpenSSH (≥ 8.4) can drive the password prompt from a helper program, so the password flows through the process env only — never a file, never argv.

1. Generate a key (if none):

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
   

2. Create an askpass helper that echoes the password from an env var (the helper itself stores NO secret):

   printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
   chmod 700 /tmp/askpass.sh
   

3. Copy the pubkey, forcing the askpass path and detaching from the TTY:

   env DEPLOY_PW='<password>' \
       SSH_ASKPASS=/tmp/askpass.sh \
       SSH_ASKPASS_REQUIRE=force \
       setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
   

   Password via the transient env argument only.
4. Add host aliases in ~/.ssh/config so deploy scripts resolve names (HostName, User, IdentityFile, IdentitiesOnly yes).
5. Verify with ssh -o BatchMode=yes myhost 'echo OK; hostname' — BatchMode fails fast if a password would be required; a clean success proves the key works.

Gotchas: without SSH_ASKPASS_REQUIRE=force, ssh ignores SSH_ASKPASS when a TTY is present and prompts interactively (hangs in a non-interactive tool); setsid detaches the controlling terminal — required on older OpenSSH even with force; the helper must be executable (chmod 700) and print ONLY the password; IdentitiesOnly yes stops ssh offering other agent keys that could be rejected first ("Too many authentication failures"). Passwordless sudo on the target is a separate concern (NOPASSWD / FRANK_SUDO_PW).

Remote execution on a fish login shell

Running remote commands or shipping files over SSH to a host whose login shell is fish (blankpad: user blank, reachable only as mDNS FQDN blankpad.local / 192.168.0.83 — the bare hostname does not resolve from the workstation).

1. Auth: SSHPASS='<pw>' sshpass -e ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…' — password via env, never argv (ps-visible).
2. Remote exec: ssh host 'cmd' pipes the command string to the login shell (fish). Bashisms — for … do, $?, heredoc markers — fail at parse time on fish (Missing end to balance this for loop). Always wrap compound commands: ssh host 'bash -c "for b in a b c; do command -v $b || echo MISSING; done"' (escape inner $).
3. File shipping: never combine cat > /remote/path <<EOF with a trailing bash -s in the same ssh command — cat drains stdin, so the interpreter silently runs on empty stdin and prints nothing (a silent no-op that looks like success). Ship real bytes via stdin redirect: ssh host 'cat > /remote/path' < localfile, then verify byte-truth with md5sum on both ends.
4. Exit codes: $? in an unquoted remote command is parsed by fish ($status there) — wrap in bash or drop the status echo; ssh's own exit code propagates to the caller anyway.

Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first probe died on fish loop parsing; a cat > ship silently swallowed the heredoc meant for bash -s, deploying wrong bytes until md5-compared.

Related: dankserver's login shell is ZSH — never start an unquoted word with = in an ssh one-liner (echo === aborts the whole chain).

GUI hang — remote diagnosis over SSH

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

1. Find the smoking gun — the kernel hung-task reporter already logged it:

   journalctl -b -p err --since "-2 hours" --no-pager | tail -80
   

   Look for INFO: task X blocked for more than N seconds and blocked on a semaphore likely last held by task Y:PID (modern kernels name the lock holder outright).
2. Full stacks for victim and holder:

   journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
   cat /proc/<pid>/status | head -5    # State: R (spinning) vs D (blocked)
   

   Victim D + holder R = livelock/deadlock with the lock held; voluntary_ctxt_switches in the millions on an R kthread = spinning, not progressing.
3. Hardware or software? GPU/CPU still alive, no Xid errors?

   nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
   journalctl -b -k --no-pager | grep -i xid
   

   Instant answers with no Xid ⇒ driver lock issue, not hardware.
4. Trigger window: block start = first hung message timestamp − reported seconds; then inspect user-session events around it (journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit).
5. Collateral: other D tasks (ps -eo pid,stat,comm | awk '$2 ~ /D/'), USB workers, etc.

Recoverability verdict: compositor in D state is unkillable and a spinning kernel thread holding the lock cannot be poked — the only exit is reboot: sync && systemctl reboot --force --force (double-force skips stop-jobs waiting forever on D-state PIDs). If the holder is D (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.

After diagnosing: map stack symbols to driver internals to name the exact subsystem, check upstream for the exact symbols + driver version, and propose targeted off-switches (NVreg_RegistryDwords, module params from /proc/driver/nvidia/params) rather than blanket driver swaps.

Related

• Fleet Ops Playbook — incident procedures that run over these sessions
• Remote Job Runner — long jobs that must survive session loss
• Deploy State Triage — the deploy flow these credentials serve