Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

6.1 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

SSH and Remote Access

Passwordless SSH bootstrap without sshpass/expect

Use when you must install your pubkey on remote hosts but sshpass and expect are not installed and you only have a password. Modern OpenSSH (≥ 8.4) can drive the password prompt from a helper program, so the password flows through the process env only — never a file, never argv.

Generate a key (if none):

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "host-deploy-$(date +%Y%m%d)"

Create an askpass helper that echoes the password from an env var (the helper itself stores NO secret):
```
printf '#!/bin/sh\necho "$DEPLOY_PW"\n' > /tmp/askpass.sh
chmod 700 /tmp/askpass.sh
```

Copy the pubkey, forcing the askpass path and detaching from the TTY:

env DEPLOY_PW='<password>' \
    SSH_ASKPASS=/tmp/askpass.sh \
    SSH_ASKPASS_REQUIRE=force \
    setsid ssh-copy-id -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 user@host

Password via the transient env argument only.

Add host aliases in ~/.ssh/config so deploy scripts resolve names (HostName, User, IdentityFile, IdentitiesOnly yes).
Verify with ssh -o BatchMode=yes myhost 'echo OK; hostname' — BatchMode fails fast if a password would be required; a clean success proves the key works.

Gotchas: without SSH_ASKPASS_REQUIRE=force, ssh ignores SSH_ASKPASS when a TTY is present and prompts interactively (hangs in a non-interactive tool); setsid detaches the controlling terminal — required on older OpenSSH even with force; the helper must be executable (chmod 700) and print ONLY the password; IdentitiesOnly yes stops ssh offering other agent keys that could be rejected first ("Too many authentication failures"). Passwordless sudo on the target is a separate concern (NOPASSWD / FRANK_SUDO_PW).

Running remote commands or shipping files over SSH to a host whose login shell is fish (blankpad: user blank, reachable only as mDNS FQDN blankpad.local / 192.168.0.83 — the bare hostname does not resolve from the workstation).

Auth: SSHPASS='<pw>' sshpass -e ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@host '…' — password via env, never argv (ps-visible).
Remote exec: ssh host 'cmd' pipes the command string to the login shell (fish). Bashisms — for … do, $?, heredoc markers — fail at parse time on fish (Missing end to balance this for loop). Always wrap compound commands: ssh host 'bash -c "for b in a b c; do command -v $b || echo MISSING; done"' (escape inner $).
File shipping: never combine cat > /remote/path <<EOF with a trailing bash -s in the same ssh command — cat drains stdin, so the interpreter silently runs on empty stdin and prints nothing (a silent no-op that looks like success). Ship real bytes via stdin redirect: ssh host 'cat > /remote/path' < localfile, then verify byte-truth with md5sum on both ends.
Exit codes: $? in an unquoted remote command is parsed by fish ($status there) — wrap in bash or drop the status echo; ssh's own exit code propagates to the caller anyway.

Both traps fired during the 2026-09-09 blankpad dotfiles deploy: the first probe died on fish loop parsing; a cat > ship silently swallowed the heredoc meant for bash -s, deploying wrong bytes until md5-compared.

Related: dankserver's login shell is ZSH — never start an unquoted word with = in an ssh one-liner (echo === aborts the whole chain).

GUI hang — remote diagnosis over SSH

Use when the desktop is frozen, VT switches do nothing, but SSH still works.

Find the smoking gun — the kernel hung-task reporter already logged it:
```
journalctl -b -p err --since "-2 hours" --no-pager | tail -80
```
Look for INFO: task X blocked for more than N seconds and blocked on a semaphore likely last held by task Y:PID (modern kernels name the lock holder outright).
Full stacks for victim and holder:
```
journalctl -b -k --since "<hang minute -5>" --no-pager | tail -120
cat /proc/<pid>/status | head -5    # State: R (spinning) vs D (blocked)
```
Victim D + holder R = livelock/deadlock with the lock held; voluntary_ctxt_switches in the millions on an R kthread = spinning, not progressing.

Hardware or software? GPU/CPU still alive, no Xid errors?

nvidia-smi --query-gpu=driver_version,pstate,temperature.gpu,clocks_event_reasons.active --format=csv
journalctl -b -k --no-pager | grep -i xid

Instant answers with no Xid ⇒ driver lock issue, not hardware.

Trigger window: block start = first hung message timestamp − reported seconds; then inspect user-session events around it (journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit).
Collateral: other D tasks (ps -eo pid,stat,comm | awk '$2 ~ /D/'), USB workers, etc.

Recoverability verdict: compositor in D state is unkillable and a spinning kernel thread holding the lock cannot be poked — the only exit is reboot: sync && systemctl reboot --force --force (double-force skips stop-jobs waiting forever on D-state PIDs). If the holder is D (sleeping on something that can complete, e.g. storage), recovery may be possible by clearing that dependency first.

After diagnosing: map stack symbols to driver internals to name the exact subsystem, check upstream for the exact symbols + driver version, and propose targeted off-switches (NVreg_RegistryDwords, module params from /proc/driver/nvidia/params) rather than blanket driver swaps.

Fleet Ops Playbook — incident procedures that run over these sessions
Remote Job Runner — long jobs that must survive session loss
Deploy State Triage — the deploy flow these credentials serve

6.1 KiB Raw Blame History Unescape Escape

SSH and Remote Access

Passwordless SSH bootstrap without sshpass/expect

Remote execution on a fish login shell

GUI hang — remote diagnosis over SSH

Related

6.1 KiB

Raw Blame History