Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
131 lines
6.1 KiB
Markdown
131 lines
6.1 KiB
Markdown
---
|
||
type: reference
|
||
title: SSH and Remote Access
|
||
description: "Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage."
|
||
tags: [ssh, remote, fish, gui, diagnosis]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# SSH and Remote Access
|
||
|
||
## Passwordless SSH bootstrap without sshpass/expect
|
||
|
||
Use when you must install your pubkey on remote hosts but `sshpass` and
|
||
`expect` are not installed and you only have a password. Modern OpenSSH
|
||
(≥ 8.4) can drive the password prompt from a helper program, so the password
|
||
flows through the **process env only** — never a file, never argv.
|
||
|
||
1. **Generate a key** (if none):
|
||
```bash
|
||
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"
|
||
```
|
||
2. **Create an askpass helper** that echoes the password from an env var (the
|
||
helper itself stores NO secret):
|
||
```bash
|
||
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
|
||
chmod 700 /tmp/askpass.sh
|
||
```
|
||
3. **Copy the pubkey**, forcing the askpass path and detaching from the TTY:
|
||
```bash
|
||
env DEPLOY_PW='<password>' \
|
||
SSH_ASKPASS=/tmp/askpass.sh \
|
||
SSH_ASKPASS_REQUIRE=force \
|
||
setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host
|
||
```
|
||
Password via the transient `env` argument only.
|
||
4. **Add host aliases** in `~/.ssh/config` so deploy scripts resolve names
|
||
(`HostName`, `User`, `IdentityFile`, `IdentitiesOnly yes`).
|
||
5. **Verify** with `ssh -o BatchMode=yes myhost 'echo OK; hostname'` —
|
||
BatchMode fails fast if a password would be required; a clean success
|
||
proves the key works.
|
||
|
||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||
TTY is present and prompts interactively (hangs in a non-interactive tool)
|
||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||
be rejected first ("Too many authentication failures"). Passwordless **sudo**
|
||
on the target is a separate concern (NOPASSWD / `FRANK_SUDO_PW`).
|
||
|
||
## Remote execution on a fish login shell
|
||
|
||
Running remote commands or shipping files over SSH to a host whose login
|
||
shell is **fish** (blankpad: user `blank`, reachable only as mDNS FQDN
|
||
`blankpad.local` / 192.168.0.83 — the bare hostname does not resolve from the
|
||
workstation).
|
||
|
||
1. **Auth**: `SSHPASS='<pw>' sshpass -e ssh -o
|
||
PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…'`
|
||
— password via env, never argv (ps-visible).
|
||
2. **Remote exec**: `ssh host 'cmd'` pipes the command string to the login
|
||
shell (fish). Bashisms — `for … do`, `$?`, heredoc markers — fail at
|
||
*parse* time on fish (`Missing end to balance this for loop`). Always wrap
|
||
compound commands: `ssh host 'bash -c "for b in a b c; do command -v $b ||
|
||
echo MISSING; done"'` (escape inner `$`).
|
||
3. **File shipping**: never combine `cat > /remote/path <<EOF` with a
|
||
trailing `bash -s` in the same ssh command — `cat` drains stdin, so the
|
||
interpreter silently runs on empty stdin and prints nothing (a silent
|
||
no-op that *looks* like success). Ship real bytes via stdin redirect:
|
||
`ssh host 'cat > /remote/path' < localfile`, then verify byte-truth with
|
||
`md5sum` on both ends.
|
||
4. **Exit codes**: `$?` in an unquoted remote command is parsed by fish
|
||
(`$status` there) — wrap in bash or drop the status echo; ssh's own exit
|
||
code propagates to the caller anyway.
|
||
|
||
Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first
|
||
probe died on fish loop parsing; a `cat >` ship silently swallowed the
|
||
heredoc meant for `bash -s`, deploying wrong bytes until md5-compared.
|
||
|
||
Related: dankserver's login shell is ZSH — never start an unquoted word with
|
||
`=` in an ssh one-liner (`echo ===` aborts the whole chain).
|
||
|
||
## GUI hang — remote diagnosis over SSH
|
||
|
||
Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||
|
||
1. **Find the smoking gun** — the kernel hung-task reporter already logged
|
||
it:
|
||
```bash
|
||
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
|
||
```
|
||
Look for `INFO: task X blocked for more than N seconds` and `blocked on a
|
||
semaphore likely last held by task Y:PID` (modern kernels name the lock
|
||
holder outright).
|
||
2. **Full stacks** for victim and holder:
|
||
```bash
|
||
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
|
||
cat /proc/<pid>/status | head -5 # State: R (spinning) vs D (blocked)
|
||
```
|
||
Victim `D` + holder `R` = livelock/deadlock with the lock held;
|
||
`voluntary_ctxt_switches` in the millions on an `R` kthread = spinning,
|
||
not progressing.
|
||
3. **Hardware or software?** GPU/CPU still alive, no Xid errors?
|
||
```bash
|
||
nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
|
||
journalctl -b -k --no-pager | grep -i xid
|
||
```
|
||
Instant answers with no Xid ⇒ driver lock issue, not hardware.
|
||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||
seconds; then inspect user-session events around it
|
||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`)
|
||
USB workers, etc.
|
||
|
||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||
spinning kernel thread holding the lock cannot be poked — **the only exit is
|
||
reboot**: `sync && systemctl reboot --force --force` (double-force skips
|
||
stop-jobs waiting forever on D-state PIDs). If the holder is `D` (sleeping on
|
||
something that can complete, e.g. storage), recovery may be possible by
|
||
clearing that dependency first.
|
||
|
||
After diagnosing: map stack symbols to driver internals to name the exact
|
||
subsystem, check upstream for the exact symbols + driver version, and propose
|
||
targeted off-switches (`NVreg_RegistryDwords`, module params from
|
||
`/proc/driver/nvidia/params`) rather than blanket driver swaps.
|
||
|
||
## Related
|
||
|
||
- [[Fleet Ops Playbook]] — incident procedures that run over these sessions
|
||
- [[Remote Job Runner]] — long jobs that must survive session loss
|
||
- [[Deploy State Triage]] — the deploy flow these credentials serve
|