command-center/agent/managed-skills/sftp-client-unreachable-triage/SKILL.md

name, description

name	description
sftp-client-unreachable-triage	Diagnose "SFTP/SSH from box A to box B doesn't work" when box A is unreachable from the fleet: exonerate the server with a log+key census, then discriminate sleep vs firewall vs name-resolution vs missing key from outside.

Triage a client-side SFTP/SSH failure (client box unreachable)

Use when Frank reports "FileZilla/SFTP/scp from to isn't working" and the client box is not reachable from the fleet. Produces a ranked, evidence-backed verdict without access to the client. Read-only probes only; no restarts, no config writes.

Order of operations (each step can end the investigation)

1. Server-side exoneration — gather before theorising

ssh -o BatchMode=yes <server> 'sudo -n sshd -T | grep -Ei "^(port|listenaddress|pubkeyauth|passwordauth|kbdinteractive|allowusers|subsystem|authorizedkeysfile)"'
ssh -o BatchMode=yes <server> 'ls -l /usr/lib/openssh/sftp-server; sudo -n ls -l /etc/ssh/ssh_host_*key.pub'   # host-key mtime = known-hosts churn check
ssh -o BatchMode=yes <server> 'sudo -n iptables -S | head; sudo -n nft list ruleset | head -20'               # INPUT ACCEPT = no firewall

Host keys unchanged for years ⇒ FileZilla cannot be tripping a host-key warning.

2. Censuses that turn "maybe" into "never" (the payload of this skill)

# keys trusted by the server, by comment
ssh <server> 'ssh-keygen -lf ~/.ssh/authorized_keys'

# every IP that ever authenticated, and with which key
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
  | grep -oE "Accepted publickey for [a-z]+ from [0-9.]+" | awk "{print \$NF}" | sort | uniq -c'
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
  | grep -oE "ED25519 SHA256:[A-Za-z0-9+/=]+" | sort | uniq -c'

# did the client IP EVER appear, anywhere?
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-45 days" | grep -c "<client-ip>"'
ssh <server> 'sudo -n zgrep -h "<client-ip>" /var/log/auth.log* | grep -v sudo | tail'

Count 0 across journal and rotated logs ⇒ no TCP ever established ⇒ the failure is DNS, client egress, or a sleeping host — not credentials. Say so explicitly, because "permission denied" and "connection timed out" have different fixes.

3. Is the client even at the IP everyone believes? (DHCP drift)

avahi-resolve -n <Client>.local    # mDNS truth
ip neigh | grep -E '\.(old|new) '  # FAILED vs REACHABLE
nmap -sn -n 192.168.0.0/24         # works unprivileged; arp-scan needs root

macOS Private Wi-Fi Address rotation changes the MAC ⇒ new DHCP lease ⇒ every hardcoded IP for that box rots. Check both old and new addresses for mentions in /var/log/auth.log* and journalctl.

4. Discriminate sleep vs firewall (ARP/mDNS alive, everything else dead)

• ARP replies + live mDNS but zero ICMP and zero TCP on a spread of ports (22, 80, 445, 548, 5900, 8080, 5000, 7000, 3283) from multiple hosts including a wired one: probe from a wired box on the same L2 — that also rules out Wi-Fi AP isolation.
• Same result ⇒ either the host is asleep (Bonjour sleep proxy / AP answers ARP+mDNS on its behalf) or a host packet filter (macOS "Block all incoming", LuLu/Little Snitch, ProtonVPN kill switch) is dropping it.
• Do not overclaim which. State both, and note the built-in macOS firewall only blocks inbound — so if Frank is actively using that Mac, a built-in-firewall verdict cannot explain an outbound SFTP failure; a third-party filter/VPN or DNS is then the cause.

5. Name resolution (the silent failure)

getent hosts <server> <server>.local <server>.<domain>
dig +short <server> @<pihole-ip>

Traps seen on this fleet:

• Bare dankserver resolves nowhere — no Pi-hole custom record, gateway DNS refuses client queries. Only mDNS <host>.local, which may advertise a link-local IPv6 only.
• dankserver.franksdank.cloud resolves to Caddy (192.168.0.143), not the SSH box — a wildcard vhost. Never suggest a *.franksdank.cloud name for SSH/SFTP.
• FileZilla ignores ~/.ssh/config — Host dankserver aliases do not apply; the Host field needs a literal IP that resolves.

6. Fix recipe handed to the user (client side)

ping -c2 <server-ip> && nc -vz <server-ip> 22
ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519 -C "<client-name>"
ssh-copy-id -i ~/.ssh/id_ed25519.pub dank@<server-ip>
sftp -i ~/.ssh/id_ed25519 dank@<server-ip>

FileZilla: Protocol SFTP, Host literal IP, Port 22, Logon Type Key file → /Users/dank/.ssh/id_ed25519 (or Normal + password). The "Encryption" dropdown is FTP-only noise for SFTP sites.

7. Report collateral you proved, don't guess it

Dead client IP ⇒ grep the fleet for hardcoded references and observe the breakage:

grep -rn --exclude-dir=.git "<old-ip>" ~/Projects/serverAppConfig | wc -l
curl -sk -o /dev/null -w '%{http_code}\n' --resolve "radarr.franksdank.cloud:443:<caddy-ip>" https://radarr.franksdank.cloud/

(-H Host: alone gives 000 — curl sends the IP as TLS SNI and the handshake fails. Use --resolve to keep SNI correct before concluding anything about a vhost.)

Laws

• Zero packets at the server beats any credential theory; log census first, key census second.
• ARP/mDNS answering is not proof a host is awake; the AP or a sleep proxy can answer.
• Probe from a wired host before blaming Wi-Fi isolation.
• A rotating private Wi-Fi MAC will re-break every hardcoded IP: recommend a DHCP reservation as the fix, not 40 file edits.