Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
131 lines
6.1 KiB
Markdown
131 lines
6.1 KiB
Markdown
---
|
||
type: reference
|
||
title: SSH and Remote Access
|
||
description: "Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage."
|
||
tags: [ssh, remote, fish, gui, diagnosis]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# SSH and Remote Access
|
||
|
||
## Passwordless SSH bootstrap without sshpass/expect
|
||
|
||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||
flows through the **process env only** — never a file, never argv.
|
||
|
||
1. **Generate a key** (if none):
|
||
```bash
|
||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||
```
|
||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||
helper itself stores NO secret):
|
||
```bash
|
||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||
chmod 700 /tmp/askpass.sh
|
||
```
|
||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||
```bash
|
||
env DEPLOY_PW='<password>' \
|
||
SSH_ASKPASS=/tmp/askpass.sh \
|
||
SSH_ASKPASS_REQUIRE=force \
|
||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||
```
|
||
Password via the transient `env` argument only.
|
||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||
BatchMode fails fast if a password would be required; a clean success
|
||
proves the key works.
|
||
|
||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||
|
||
## Remote execution on a fish login shell
|
||
|
||
Running remote commands or shipping files over SSH to a host whose login
|
||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||
workstation).
|
||
|
||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||
— password via env, never argv (ps-visible).
|
||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||
echo MISSING; done"'` (escape inner `$`).
|
||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||
`md5sum` on both ends.
|
||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||
code propagates to the caller anyway.
|
||
|
||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||
|
||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||
|
||
## GUI hang — remote diagnosis over SSH
|
||
|
||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||
|
||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||
it:
|
||
```bash
|
||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||
```
|
||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||
holder outright).
|
||
2. **Full stacks** for victim and holder:
|
||
```bash
|
||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||
```
|
||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||
not progressing.
|
||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||
```bash
|
||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||
journalctl -b -k --no-pager | grep -i xid
|
||
```
|
||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||
seconds; then inspect user-session events around it
|
||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||
USB workers, etc.
|
||
|
||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||
something that can complete, e.g. storage), recovery may be possible by
|
||
clearing that dependency first.
|
||
|
||
After diagnosing: map stack symbols to driver internals to name the exact
|
||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||
|
||
## Related
|
||
|
||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|