Compare commits

...
22 changed files with 587 additions and 15 deletions

No files matched your search

@@ -0,0 +1,53 @@
---
name: live-checkout-landing-chain
description: "Prove whether a merged change is actually deployed in ~/.omp: walk origin/main vs the checked-out main commit vs the worktree, then fast-forward in place — git refuses fetch into a checked-out branch, and a running harness caches its config."
---
## When to use
The user says a merged/committed change is "not there" (`/switch` can't see a model, a doc is missing, a config edit is invisible) while the repo work itself is green. In a repo where **the checkout is the deployment** (`dank/command-center` at `~/.omp`), "merged" and "deployed" are different commits — measure both before believing either claim.
## The ladder — read all four levels, in order, before answering
```bash
cd ~/.omp && git rev-parse HEAD origin/main # 1+2: deployment vs forge
git ls-tree --name-only HEAD # is the file even in HEAD?
git worktree list # 3: which worktree am I in vs main?
```
1. `origin/main` — the forge truth (`gh`/`tea` absent; Gitea API or a fetch).
2. `/home/blank/.omp` HEAD — **what the running agent actually reads.** Merged PR at `3eb1ada` while this sat at `53025a6` is the whole bug.
3. The feature/session worktree — where the edit was authored; its files can be correct while main is stale, which is why the author sees the change and the user does not.
4. The running process — `models.yml` / `config.yml` are read at startup and cached for the process lifetime; an open session needs a restart even after level 2 is correct.
Belt-and-braces when the answer hinges on file contents, not just commit ids: parse the live file (`python3 -c "import yaml; ..."` for `agent/models.yml`, or grep the doc) rather than inferring from `git log`.
## Landing main without touching the tree
`git fetch origin main:main` **fails**: `fatal: refusing to fetch into branch 'refs/heads/main' checked out at '/home/blank/.omp'`. The fetch-into-ref trick only works for branches that are *not* checked out anywhere.
For a checked-out main, prove then fast-forward:
```bash
git merge-base --is-ancestor main origin/main && echo "ff possible"
git merge --ff-only origin/main # no rebase — Frank dislikes history rewrites
```
`--ff-only` is the safe verb on a dirty main checkout: tracked-file commits (the live-sync tier) and runtime dirt (`agent/memories/*/state.json`, `dotfiles/**`) survive untouched, and it cannot rewrite history. Confirm the dirt is still there afterwards — if it vanished, something else clobbered the tree.
## The heartbeat is not instant
`scripts/live-sync.sh` (systemd user timer, hourly) is what normally advances main. Check its cadence rather than assuming it ran:
```bash
systemctl --user status live-sync.timer --no-pager # "Trigger: …" = next beat
systemctl --user status live-sync.service --no-pager # "since …" = last run
```
A merge made minutes after the last beat waits until the next one; fast-forwarding by hand is legitimate (idempotent with the script) and is the answer when the user wants it now. Never `--no-verify`, never force-push.
## Do not confuse with
- `omp-worktree-path-gotchas` — edits landing in the *wrong* worktree (cwd drift), not a stale main.
- `dotfiles-deployment-truth-audit` — classifies which files are tracked vs runtime state.
- `serverappconfigs-vault-page-registration` / `gitea-merge-form-encoded` — the PR mechanics upstream of this question.
@@ -0,0 +1,86 @@
---
name: omp-provider-row-sources
description: "Add a provider/model row to ~/.omp/agent/models.yml by reading the harness's own catalog tables (provider name, api, window, cost, compat) instead of guessing or web-searching, then prove it with a scratch-HOME call."
---
# Scope
Adding (or correcting) a provider/model row in `~/.omp/agent/models.yml` for a
provider the harness may already know: an upstream vendor API (DeepSeek, OpenAI,
Anthropic, Moonshot, …) or a new local endpoint. The rule this skill exists for:
**almost every field you are about to research on the web already exists in the
installed bundle** — research there first, and the web only confirms it.
Canonical install root (resolve the symlink first if the repo lives under a
worktree): `/home/blank/.bun/install/global/node_modules/@oh-my-pi/`.
# The three tables that own the facts
| You want | Read | Why |
|---|---|---|
| The legal **provider name**, its URL marker, and which quirks the harness gates on it | `pi-catalog/src/hosts.ts` (minified in `dist/cli.js` as the `VIn` map; consumed by `pi-catalog/src/compat/resolve.ts`) | The provider key is a **lookup key, not a display name**. e.g. `deepseekDirect: { providers: ["deepseek"], urlMarkers: ["api.deepseek.com"] }`, `anthropic: { providers: ["anthropic"], urlMarkers: ["api.anthropic.com"] }`. A label matching neither loses the behaviour set silently (the `max_tokens` field name, the thinking `extraBody`, the thinking-disable mode). |
| Model rows: ids, `contextWindow`, `maxTokens`, `cost` rates, per-model `compat` | `pi-catalog/src/provider-models/<api>.ts` — `openai-compat.ts`, `anthropic-messages.ts`, …; factory helpers `openAiCompletionsDescriptor(...)` / `anthropicMessagesDescriptor(...)` | Rows are fetched from the provider's models-dev key, so the bundle carries the same numbers the vendor publishes (and often rates the vendor page hides). The descriptor's `compat:` block is the authored truth for that provider's transport shape — copy it, never simplify it. |
| The **schema** itself (which keys are legal on a provider vs a model, and which are required) | `pi-coding-agent/src/config/models-config-schema-bundle.ts` (and the identical `dist/` copy) | Provider level and model level have *different* field sets. Cheaper and more correct than reading the minified bundle. |
Also in `pi-coding-agent/src/config/`: `model-registry.ts` (how `models.yml`
merges onto the built-in catalog — `#mergeCustomModels` → `mergeByModelKey`),
`resolve-config-value.ts` (how `apiKey` values resolve).
# Two traps this table pays for
- **Provider-level `api` only accepts chat/judge transports.** Naming a runner API
(`openai-speech`, `openai-images`, `openai-embeddings`) at provider level fails
whole-file validation and disables *every* custom provider, not just yours
(`omp models` prints "models.yml validation failed", lists only built-ins).
Runner api belongs on the **model** entry; see the managed skill
`omp-models-yml-runner-api-placement`.
- **A `models.yml` entry is an overlay, not a replacement.** Adding one model to a
provider the catalog already ships does not remove the provider's other rows —
`omp models <provider>` will show yours (yours wins on shared fields, including
`cost` and `contextWindow`) plus the catalog rows. Same-id rows patch with the
`replace` transport policy; a row that omits `contextWindow` inherits the
bundled row's window and its policy tiers.
# Secret handling (non-prose tracked file)
Never inline the credential. `apiKey` resolution order is: leading `!` → command
(`stdout` trimmed, cached for the process lifetime) → exact env-var name → literal.
So write `apiKey: "!cat ~/.config/secrets/<name>_api_key"` with mode 600.
- A bare `sk-…` trips the pre-commit **shape** gate (every file type).
- An unquoted `!` is a YAML tag — quote it.
- A quoted command value also dodges the *assignment* gate: the gate's value
charset rejects `~` and spaces, so only a literal key matches.
# Procedure
1. Read the three tables above for the provider you are adding. Take the exact
provider name, base URL, api, and compat from them; take `contextWindow` /
`maxTokens` / `cost` from the model row and cite the source in a comment.
2. Write the secret to `~/.config/secrets/<name>_api_key`, `chmod 600`.
3. Edit `models.yml`; for a repo-tracked config, follow that repo's own
branch/log/index rules (in `dank/command-center`: branch → issue → PR).
4. Verify with a **scratch HOME**, because `models.yml` loads at process start
and a live session keeps the old registry:
```bash
mkdir -p /tmp/ompx/.omp/agent /tmp/ompx/.config
cp agent/models.yml agent/config.yml /tmp/ompx/.omp/agent/
ln -sfn ~/.config/secrets /tmp/ompx/.config/secrets # ~ in a !cmd value = $HOME
HOME=/tmp/ompx omp models --no-extensions | grep -c 'validation failed' # must be 0
HOME=/tmp/ompx omp --model '<provider>/<id>:off' --tools bash --no-session \
--no-skills --no-rules --no-extensions --no-pty --no-lsp --no-title \
--cwd /tmp -p 'Reply with exactly the word OK and nothing else.'
```
Then the **negative control**, the step that separates a real auth success from
a cached/built-in answer: `sed` the `baseUrl` to a dead port in the scratch copy
and repeat — it must hang or error, not answer. Then `rm -rf /tmp/ompx` (it
holds endpoint URLs).
# Voice notes
- Prefer the bundled schema file over grepping `dist/cli.js`; grep the bundle only
when the source and the bundle disagree, or for a runtime default the schema does
not state (e.g. `reasoningDisableMode` per provider family).
- `cost` is optional in the schema. Omitting it makes the status line report `$0`
forever; one rate per direction is all the schema models, so a peak/off-peak
vendor table can only carry the peak value.
@@ -0,0 +1,91 @@
---
name: quant-pipeline-adversarial-review
description: "Review or audit the serverAppConfigs quantization campaign pipeline (convert-calib-kldref-score plus corpus builder/rechunker/pusher, fp8 stacker) for defects that silently waste GPU-hours or produce a number that is not the claimed number."
---
# Adversarial review of the quant campaign pipeline
Friction-earned 2026-10-29 across `scripts/convert-qwen38-w5nl64.sh`,
`run-qwen38-bf16.sh`, `make-calib-corpus.py`, `rechunk-kld-corpus.py`,
`calib-push.py`, `stack-qwen38-official-fp8.py`. Every class below was a real
defect in that tree. Review code, not vibes: quote the line, name the cost.
## Files in scope (one campaign = this set)
| Stage | File | Owns |
|---|---|---|
| corpus | `make-calib-corpus.py` | calib + kld JSONL, manifest |
| corpus | `rechunk-kld-corpus.py` | windows that fit the container's KV window |
| convert | `convert-<model>.sh` | plan → artifact per arm |
| tape | `run-qwen38-bf16.sh` | the rig's argv, mount, trap, gates |
| drive | `calib-push.py` | prefill traffic into the tap |
| fp8 path | `stack-qwen38-official-fp8.py` | expert stacking + PLE merge |
## Defect classes to hunt, in cost order
1. **Claimed identity that is only a length check.** A tool that writes TEXT back
out (`tok.decode(window)`) and asserts only counts is not proving token
identity — decode→encode can merge/split. Fix: re-encode each window and
compare id lists; refuse on mismatch. Same shape: an assertion on "N tokens
in, N tokens out" that counts windows rather than re-encoded text.
2. **A loop that cannot finish and cannot notice.** Drivers that re-push a corpus
`--max-rounds` times without measuring the sink. Compare the tap/output dir's
byte total per round and abort on zero growth; abort on zero successes; abort
on zero prompts. Without this the loss is silent for hours.
3. **`set -u` argument holes.** `--flag) VAR="$2"; shift 2` dies with "unbound
variable" when the flag is last. Guard `[ "$#" -ge 2 ]` in every such arm.
4. **Restore paths that leave the GPU holder alive.** `--rm` only fires when
podman itself exits, so a killed run leaves conmon holding the cards while
`systemctl is-active` says `active`. `restore()` MUST `podman rm -f "$C"`
before starting production. Prove with `podman ps` after a `kill`.
5. **Gates that run after the expensive phase.** A leftover container name, an
existing output, a non-writable tap dir, an unset figure — check them before
the plan/boot, not when podman/sed fails mid-run.
6. **Zero-exit with no artifact.** After `rc=0`, assert the output file exists
before `stat`/arithmetic on it.
7. **`assert` in a verifier.** `python -O` deletes it silently. Use `raise`.
8. **Warning where the code's own assumption breaks.** e.g. a 128-lane pack fed
an HD that is not a multiple of 128 — warning + continue writes a wrong table.
9. **Re-tokenization in loops.** A pool cycled 40× re-tokenizing each fragment
per pass; and a manifest recomputing a total by re-encoding the whole corpus.
Cache counts once, accumulate during generation.
10. **Substring tests standing in for key tests.** `"content" in line` before
`json.loads(line)["content"]` crashes on any list-shaped `.json`.
11. **Path guards compared as strings.** `A/../A/x` defeats an equality check;
add `realpath -m` on both sides.
12. **Shellcheck's honest false positive**: single-quoted `$CALIB` passed to a
converter that expands it from the container env is correct (SC2016). Say so
in the PR; do not "fix" it.
## Procedure
1. Inventory: `wc -l` the scripts + recipes; read each **completely** (ranges,
not whole files, but every line). Recipes carry the numbers scripts assert.
2. Read the tests that pin the argv (`tests/qwen38_reference.bats`'s `SHAPE`
fixture, `tests/qwen38_campaign.bats`) — they are the contract you must not
break, and the first place a stale number hides.
3. Mechanical gates:
```bash
python3 -m py_compile scripts/*.py
bash -n scripts/<each>.sh
shellcheck -S warning scripts/<each>.sh # never pass .py to it
./tests/run.sh
```
4. Fix the real ones only; keep diffs surgical; re-run the suite after the batch.
5. Smoke the changed path for real — smoke beats the suite:
```bash
# run a local script on the server's data via the corpus venv, no checkout needed
ssh dankserver '/home/dank/venvs/radcorpus/bin/python - --corpus <corpus> --out /tmp/<x> --force' \
< scripts/rechunk-kld-corpus.py
```
Import-level helpers can be exercised with
`importlib.util.spec_from_file_location` (e.g. a new `tap_bytes()`: empty→0,
one file→its size, absent→0).
6. Stacked PRs: a review commit goes on its own branch off the open one; a diff
that includes the parent commit is a stacked PR, not a mistake — say which PR
merges first in the body.
## Report
One table of defect → cost → fix. Name the failure the defect produced in
practice (hours wasted, wrong number), because that is what makes it reviewable.
@@ -0,0 +1,71 @@
---
name: radiance-calibration-tap-shape
description: "Run and measure a radiance GPTQ calibration tap on a disk-tiered BF16 rig — n-gram tier, step sized to the document, tap marker gates, and reading the real rate from /stats instead of the log."
---
# Running and measuring a radiance calibration tap
A calibration tap (`RADIANCE_CALIB_DIR` + `RADIANCE_CALIB_TOKENS`) boots a BF16 reference container
on the serving cards under `--weights-disk-tier` and prefills a corpus until the pooled token floor
is reached, writing per-expert Grams for GPTQ. It is the one campaign phase whose cost is dominated
by **disk read volume**, not by compute, and the shape that serves production is the wrong shape for
it.
## The four things that decide the tap's wall time
1. **N-gram table tier.** On the BF16 payload the table is tens of GiB and `--ngram-placement ram`
PINS it. Beside a full host pool that leaves a few GiB of RAM and the boot dies at
`startup stopped: out of memory` while the same pool and slot count had already booted the
reference recording. Calib only needs the routed weights prefilled, so pass
`--ngram-placement disk` for calib and keep `ram` for the KL modes — their numbers are only
comparable at an identical shape.
2. **Step ≥ document length.** A document cut into N steps re-streams each layer's routed expert
union N times from the container. With `--max-num-batched-tokens` below the document length the
tap runs several times slower than with one wide step, and the KV pool does not change at all
(`kv_tokens_total` is unaffected). This is legal for calib and NOT for a scoring boot: scoring
numbers are only comparable at one shape, so the step is calib's own knob.
3. **KV pool, not slots.** When the pool holds barely one document, extra `--max-num-seqs` only
queue (and preempt) requests; raising slots does not raise throughput.
4. **The token floor is pooled, not per round.** One corpus pass may be smaller than the floor, so a
round that seals no bytes is NORMAL — a driver that treats a flat round as failure aborts one
round early. Fail on a tap that *loses* bytes; let "never sealed" fail at max rounds.
## Never push a tap that belongs to no rig
A tap dir is a directory; production is also an endpoint on the same port. Without proof, a pusher
will happily feed millions of tokens into whatever answers and record nothing.
- The rig writes a marker into the tap dir at boot (`container=`, `image=`, `pid=`), retires it
when its boot exits non-zero, and the pusher refuses unless the marker exists, names the
container it targets, and that container is running.
- The window script stops on a failed phase instead of falling through its health wait, and banks a
large reference artifact only when both phases exit 0 and the tap holds files.
## Measuring the rate (do not read it off the log)
The campaign log is `tee`'d and block-buffered, so its request count lags by minutes.
```bash
curl -s http://127.0.0.1:7770/stats | python3 -c 'import json,sys; d=json.load(sys.stdin); \
print({k: d[k] for k in ("uptime_s","prefill_tps","steps_prefill","total_requests","preemptions","kv_tokens_total")})'
```
`prefill_tps` is CUMULATIVE over uptime — `tokens ≈ prefill_tps × uptime_s`, so take two samples a
couple of minutes apart and difference them. `steps_prefill / total_requests` proves whether a
document is one step or many. Discard the first minutes after boot (warmup and the first requests
are not the steady state).
## Sizing the tap before paying for it
Probe one 5-minute sample at the candidate step, compute the ratio against the inherited shape, and
choose the floor from the measured rate — not from an earlier campaign's runtime on a different
corpus and shape. Then set `RADIANCE_CALIB_TOKENS` (via the runner's env knob) and launch ONE run;
if the probe already beat the bar, that probe run IS the real run and needs no restart.
## Deliverables that make it unattended
- a gated window script: boot → health with a hard timeout → push → stop → restore the service →
bank the reference artifact only on success;
- a pusher that is marker-gated and progress-aware;
- a ledger line recording the shape used and the target chosen, so the next session does not
re-derive it.
@@ -0,0 +1,68 @@
---
name: radiance-disk-tier-throughput-census
description: "Measure a radiance rig's real prefill rate and prove whether the bottleneck is disk, CPU or shape — for calibration taps, KLD recording or score boots; use when a rate looks wrong, an ETA must be quoted, or someone asks about CPU allocation."
---
# Census a radiance rig's throughput without lying to yourself
Ask three questions in order: what is the rate, what is the binding resource, and does the shape
explain it. Every number below comes from a kernel or engine counter — never from a gauge.
## 1. The rate: counters, never `prefill_tps`
`/stats` on :7770 exposes `prefill_tps`, and it is a **rolling gauge**. Reading 69.5 → 149.1 across
120 s and multiplying by `uptime_s` produced a bogus "320 tok/s / 10.4 h" verdict that had to be
retracted (measured 2026-10-29). Derive the rate from counters over a ≥240 s window:
```bash
python3 - <<'PY'
import json, time, urllib.request
def stats(): return json.load(urllib.request.urlopen("http://127.0.0.1:7770/stats", timeout=10))
a = stats(); t0 = time.time(); time.sleep(240); b = stats(); t1 = time.time(); dt = t1 - t0
docs = b["total_requests"] - a["total_requests"]
steps = b["steps_prefill"] - a["steps_prefill"]
MEAN = 4238.0 # tokens/doc, from the corpus: total_tokens / documents
print("docs %d in %.0fs -> %.0f tok/s | steps/doc %.2f" % (docs, dt, docs * MEAN / dt, steps / docs))
PY
```
`steps/doc` is the read-amplification signal: a document cut into N prefill steps re-streams each
layer's routed expert union N times when the weights are on the disk tier, so N > 1 is leaked time.
## 2. The binding resource: diskstats and the idle/iowait split
```bash
# per-device read bandwidth (parent device, not the partition)
awk '{print $3, $6*512}' /proc/diskstats | grep -E "^nvme"
# host busy = total - idle - iowait (summing /proc/stat's first 8 fields WITHOUT subtracting is
# the classic error: the sum already covers idle, so it reads ~800% on an 8-core box)
```
Then split CPU by process: sample `/proc/<pid>/stat` fields 14+15 (utimes + stimes) twice.
Two hard readings that decide the argument:
- `radiance` compute at ~0.26 cores while the disk delivers ~3.2 GB/s and `iowait` is ~29 % of the
box ⇒ **disk-bound**. More cores or a `--cpuset-cpus` pin cannot add bytes; pinning only caps it.
- A container with `cpu_quota=0` and an empty `CpusetCpus` (check `podman inspect -f
'{{.HostConfig.CpuQuota}} {{.HostConfig.CpusetCpus}}'`) is already unrestricted — "allocate more
CPU" is a no-op, and "allocate 6 of 8" is a ceiling.
Sanity relation: tok/s × (GiB streamed per document ÷ tokens per document) should match the measured
device bandwidth. If it does not, the shape, not the hardware, is the story.
## 3. The shape: what may legally be widened
- `kldref` and `score` must keep production's `--max-num-batched-tokens` and `--max-num-seqs`: their
KL numbers are only comparable across boots at an identical shape (the audit law).
- A **calibration tap** is exempt — its output is layer Hessians, nothing compares it. Give it a step
that covers a whole document (`Q38_CALIB_MNBT` ≥ the corpus window) and read `steps/doc`: on a
7,168-token corpus, 2,048 → 8,192 moved 2.5 → 1.00 steps/doc and 94 → 113 tok/s (**1.2×**, not the
4× the step-count ratio suggests — total bytes are unchanged, so do not promise more).
- The KV pool, not the step, bounds concurrency: when `kv_tokens_total` ≈ one document, requests queue
and preempt; extra slots buy nothing.
## Reporting
Quote tok/s, steps/doc, device GB/s and the busy/iowait split together, plus the shape each number
was taken at. A rate without its shape is not a measurement; a gauge without its window is not a rate.
@@ -0,0 +1,77 @@
---
name: script-refactor-parity-vs-previous-revision
description: "Prove a refactor of sibling data-pipeline scripts is behavior-identical: run the pre-refactor revision from git HEAD beside the new one on the host that owns the venv/data, compare parsed records rather than bytes, and bootstrap the shared module so file, runpy and container paths all import it."
---
Refactoring a pipeline of sibling scripts (corpus builders, re-chunkers, pushers) whose only
tests are smokes. The test suite cannot see a moved record boundary, so parity has to be
*measured against the previous revision*.
## 1. Keep the old revision runnable
Never trust the diff. Materialize both sides where the data lives (the host with the venv,
tokenizer, parquet, corpus):
```sh
git show HEAD:scripts/<tool>.py > /tmp/qc-old/<tool>.py # pre-refactor
cp scripts/<tool>.py scripts/<newmodule>.py /tmp/qc-new/ # post, module beside it
```
`ssh host 'cat > /tmp/...'` with the bytes on stdin is enough; no repo on the target needed.
Run both with **identical seeds and flags**, into separate output dirs.
## 2. Compare records, not bytes — unless the escaping is unchanged
- If the refactor funnels writes through one helper, that helper's `ensure_ascii` choice moves
every non-ASCII character. A byte diff then reports a false regression while the rows are
identical.
- Prove it correctly: parse both sides and compare in order —
```python
a = [json.loads(l) for l in open(old) if l.strip()]
b = [json.loads(l) for l in open(new) if l.strip()]
print(a == b, hashlib.sha256(json.dumps(a, sort_keys=True).encode()).hexdigest()[:16])
```
- Where the refactor does not touch serialization (a tool that already wrote literal UTF-8),
demand the stronger proof: `cmp` + md5 equality of the artifact.
- Order matters: shuffled output with an unchanged seed proves the RNG draw sequence survived
the hoist of nested closures. Compare the *sorted* file only as a second check — permutation
equality alone hides a reordered `rng.choice`.
- Hashing the record set with `sort_keys=True` also catches dict-key drift in manifests.
## 3. Sibling-module bootstrap that survives every invocation
A new shared module must import when the tool is run three ways: `python3 scripts/tool.py`,
`runpy.run_path('/work/tool.py')` from a container mount (`-v scripts:/work:ro`), and from any
cwd. Standard preamble:
```python
_HERE = os.path.dirname(os.path.abspath(__file__)) if "__file__" in globals() else os.getcwd()
if _HERE not in sys.path:
sys.path.insert(0, _HERE)
import <module> # noqa: E402 - sibling module, same directory
```
Piping the script through `python3 -` has no `__file__` and defeats this — run tools by path and
say so in the module docstring. Verify both forms explicitly before claiming it works.
## 4. Unit-test the refusals without the heavy dependency
Hoist validation into a function so it can be driven with a stub, in-process:
```python
spec = importlib.util.spec_from_file_location("<module>", ".../<module>.py")
m = importlib.util.module_from_spec(spec); sys.modules["<module>"] = m; spec.loader.exec_module(m)
```
Registering the module in `sys.modules` under its real name *before* loading the dependent
script is required, or the script's own `import <module>` re-imports it and your stub is invisible.
A stub object with `encode`/`decode` that mangles ids past a threshold exercises a
"token identity" check in one page of output, no tokenizer file needed.
## 5. Gate order
`py_compile` every touched file → run the tool's `--help` through both invocation forms →
old-vs-new parity on the host → suite → commit. A suite that enumerates shell seams will not
notice a corpus-tool regression; the parity run is the only evidence there.
@@ -0,0 +1,100 @@
---
name: sftp-client-unreachable-triage
description: "Diagnose \"SFTP/SSH from box A to box B doesn't work\" when box A is unreachable from the fleet: exonerate the server with a log+key census, then discriminate sleep vs firewall vs name-resolution vs missing key from outside."
---
# Triage a client-side SFTP/SSH failure (client box unreachable)
Use when Frank reports "FileZilla/SFTP/scp from <client> to <server> isn't working" and the
client box is not reachable from the fleet. Produces a ranked, evidence-backed verdict
without access to the client. **Read-only probes only**; no restarts, no config writes.
## Order of operations (each step can end the investigation)
### 1. Server-side exoneration — gather before theorising
```bash
ssh -o BatchMode=yes <server> 'sudo -n sshd -T | grep -Ei "^(port|listenaddress|pubkeyauth|passwordauth|kbdinteractive|allowusers|subsystem|authorizedkeysfile)"'
ssh -o BatchMode=yes <server> 'ls -l /usr/lib/openssh/sftp-server; sudo -n ls -l /etc/ssh/ssh_host_*key.pub' # host-key mtime = known-hosts churn check
ssh -o BatchMode=yes <server> 'sudo -n iptables -S | head; sudo -n nft list ruleset | head -20' # INPUT ACCEPT = no firewall
```
Host keys unchanged for years ⇒ FileZilla cannot be tripping a host-key warning.
### 2. Censuses that turn "maybe" into "never" (the payload of this skill)
```bash
# keys trusted by the server, by comment
ssh <server> 'ssh-keygen -lf ~/.ssh/authorized_keys'
# every IP that ever authenticated, and with which key
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
| grep -oE "Accepted publickey for [a-z]+ from [0-9.]+" | awk "{print \$NF}" | sort | uniq -c'
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-90 days" \
| grep -oE "ED25519 SHA256:[A-Za-z0-9+/=]+" | sort | uniq -c'
# did the client IP EVER appear, anywhere?
ssh <server> 'sudo -n journalctl -u ssh --no-pager --since "-45 days" | grep -c "<client-ip>"'
ssh <server> 'sudo -n zgrep -h "<client-ip>" /var/log/auth.log* | grep -v sudo | tail'
```
Count 0 across journal **and** rotated logs ⇒ no TCP ever established ⇒ the failure is
DNS, client egress, or a sleeping host — **not** credentials. Say so explicitly, because
"permission denied" and "connection timed out" have different fixes.
### 3. Is the client even at the IP everyone believes? (DHCP drift)
```bash
avahi-resolve -n <Client>.local # mDNS truth
ip neigh | grep -E '\.(old|new) ' # FAILED vs REACHABLE
nmap -sn -n 192.168.0.0/24 # works unprivileged; arp-scan needs root
```
macOS **Private Wi-Fi Address** rotation changes the MAC ⇒ new DHCP lease ⇒ every
hardcoded IP for that box rots. Check both old and new addresses for mentions in
`/var/log/auth.log*` and `journalctl`.
### 4. Discriminate sleep vs firewall (ARP/mDNS alive, everything else dead)
- ARP replies + live mDNS but **zero** ICMP and **zero** TCP on a spread of ports
(22, 80, 445, 548, 5900, 8080, 5000, 7000, 3283) from **multiple** hosts including a
wired one: probe from a wired box on the same L2 — that also rules out Wi-Fi AP isolation.
- Same result ⇒ either the host is asleep (Bonjour sleep proxy / AP answers ARP+mDNS on its
behalf) **or** a host packet filter (macOS "Block all incoming", LuLu/Little Snitch,
ProtonVPN kill switch) is dropping it.
- **Do not overclaim which.** State both, and note the built-in macOS firewall only blocks
*inbound* — so if Frank is actively using that Mac, a built-in-firewall verdict cannot
explain an *outbound* SFTP failure; a third-party filter/VPN or DNS is then the cause.
### 5. Name resolution (the silent failure)
```bash
getent hosts <server> <server>.local <server>.<domain>
dig +short <server> @<pihole-ip>
```
Traps seen on this fleet:
- Bare `dankserver` resolves **nowhere** — no Pi-hole custom record, gateway DNS refuses
client queries. Only mDNS `<host>.local`, which may advertise a **link-local IPv6 only**.
- `dankserver.franksdank.cloud` resolves to **Caddy (192.168.0.143)**, not the SSH box —
a wildcard vhost. Never suggest a `*.franksdank.cloud` name for SSH/SFTP.
- **FileZilla ignores `~/.ssh/config`** — `Host dankserver` aliases do not apply; the Host
field needs a literal IP that resolves.
### 6. Fix recipe handed to the user (client side)
```bash
ping -c2 <server-ip> && nc -vz <server-ip> 22
ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519 -C "<client-name>"
ssh-copy-id -i ~/.ssh/id_ed25519.pub dank@<server-ip>
sftp -i ~/.ssh/id_ed25519 dank@<server-ip>
```
FileZilla: Protocol **SFTP**, Host literal IP, Port 22, Logon Type **Key file** →
`/Users/dank/.ssh/id_ed25519` (or Normal + password). The "Encryption" dropdown is FTP-only
noise for SFTP sites.
### 7. Report collateral you proved, don't guess it
Dead client IP ⇒ grep the fleet for hardcoded references and *observe* the breakage:
```bash
grep -rn --exclude-dir=.git "<old-ip>" ~/Projects/serverAppConfig | wc -l
curl -sk -o /dev/null -w '%{http_code}\n' --resolve "radarr.franksdank.cloud:443:<caddy-ip>" https://radarr.franksdank.cloud/
```
(`-H Host:` alone gives `000` — curl sends the IP as TLS SNI and the handshake fails. Use
`--resolve` to keep SNI correct before concluding anything about a vhost.)
## Laws
- Zero packets at the server beats any credential theory; log census first, key census second.
- ARP/mDNS answering is **not** proof a host is awake; the AP or a sleep proxy can answer.
- Probe from a wired host before blaming Wi-Fi isolation.
- A rotating private Wi-Fi MAC will re-break every hardcoded IP: recommend a DHCP
reservation as the fix, not 40 file edits.
@@ -1,6 +1,6 @@
{
"v": 1,
"lastConsolidatedAt": 1793308587833,
"lastConsolidatedAt": 1793314235443,
"lastResult": {
"at": 1791599865775,
"sessions": 1,
@@ -1,4 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1793215693649
"lastConsolidatedAt": 1793322539823
}
@@ -1,7 +1,5 @@
## Product Quality Standards
Quant quality must hold regardless of implementation choice.
## Behavioral Updates
Hourly updates on a deterministic timer.
Merge any fixes immediately.
@@ -1,8 +1,8 @@
{
"v": 1,
"lastConsolidatedAt": 1793309257223,
"lastConsolidatedAt": 1793331473998,
"lastResult": {
"at": 1793218183375,
"at": 1793315963987,
"sessions": 1,
"deltas": 1,
"model": "frank-zealot"
@@ -0,0 +1,4 @@
## Models
- Models settings include /switch to switch to it
- hdmi monitor to the left and the other to the right of eDP-1
@@ -1,4 +1,10 @@
{
"v": 1,
"lastConsolidatedAt": 1793309241992
"lastConsolidatedAt": 1793312076890,
"lastResult": {
"at": 1793312076890,
"sessions": 1,
"deltas": 1,
"model": "frank-zealot"
}
}
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1793310990286
}
@@ -0,0 +1,10 @@
{
"v": 1,
"lastConsolidatedAt": 1793315256444,
"lastResult": {
"at": 1793314927653,
"sessions": 1,
"deltas": 1,
"model": "frank-zealot"
}
}
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1793322540660
}
@@ -107,11 +107,11 @@ widget_order = [
[lockscreen_widgets.widget."lockscreen-login-box@DP-2"]
box_height = 196.0
box_width = 810.0
cx = 1719.999755859375
cy = 1221.60009765625
cx = 599.9998779296875
cy = 1628.8001708984375
output = "DP-2"
placement_height = 1440.0
placement_width = 3440.0
placement_height = 1920.0
placement_width = 1200.0
rotation = 0.0
type = "login_box"
@@ -285,13 +285,13 @@ transition = [ "fade" ]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
[wallpaper.monitors.DP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
[wallpaper.monitors.DP-2]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
[wallpaper.monitors.HDMI-A-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-gjqmel.png"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"
[wallpaper.monitors.eDP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven-mlomkm.png"