Frank
33eae71d33
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
131 lines
6.2 KiB
Markdown
131 lines
6.2 KiB
Markdown
---
|
||
type: reference
|
||
title: SSH and Remote Access
|
||
description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session.
|
||
tags: [ssh, remote, fish, gui, diagnosis]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# SSH and Remote Access
|
||
|
||
## Passwordless SSH bootstrap without sshpass/expect
|
||
|
||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||
flows through the **process env only** — never a file, never argv.
|
||
|
||
1. **Generate a key** (if none):
|
||
```bash
|
||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||
```
|
||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||
helper itself stores NO secret):
|
||
```bash
|
||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||
chmod 700 /tmp/askpass.sh
|
||
```
|
||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||
```bash
|
||
env DEPLOY_PW='<password>' \
|
||
SSH_ASKPASS=/tmp/askpass.sh \
|
||
SSH_ASKPASS_REQUIRE=force \
|
||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||
```
|
||
Password via the transient `env` argument only.
|
||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||
BatchMode fails fast if a password would be required; a clean success
|
||
proves the key works.
|
||
|
||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||
|
||
## Remote execution on a fish login shell
|
||
|
||
Running remote commands or shipping files over SSH to a host whose login
|
||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||
workstation).
|
||
|
||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||
— password via env, never argv (ps-visible).
|
||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||
echo MISSING; done"'` (escape inner `$`).
|
||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||
`md5sum` on both ends.
|
||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||
code propagates to the caller anyway.
|
||
|
||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||
|
||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||
|
||
## GUI hang — remote diagnosis over SSH
|
||
|
||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||
|
||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||
it:
|
||
```bash
|
||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||
```
|
||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||
holder outright).
|
||
2. **Full stacks** for victim and holder:
|
||
```bash
|
||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||
```
|
||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||
not progressing.
|
||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||
```bash
|
||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||
journalctl -b -k --no-pager | grep -i xid
|
||
```
|
||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||
seconds; then inspect user-session events around it
|
||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||
USB workers, etc.
|
||
|
||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||
something that can complete, e.g. storage), recovery may be possible by
|
||
clearing that dependency first.
|
||
|
||
After diagnosing: map stack symbols to driver internals to name the exact
|
||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||
|
||
## Related
|
||
|
||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|