live: heartbeat sync 2026-10-21T17:49:41-07:00 (Refs #11)

agent/managed-skills/iptables-insert-order-drop-shadows-accepts/SKILL.md

@@ -0,0 +1,32 @@
+---
+name: iptables-insert-order-drop-shadows-accepts
+description: "Diagnose or prevent a service port that times out beside a LISTENing socket because a deployer's iptables -I sequence landed a blanket DROP above its ACCEPTs; includes the sandboxed fake-iptables red/green bats pattern."
+---
+
+# iptables `-I` order trap: blanket DROP shadowing ACCEPTs
+
+## Failure signature
+- Client to a port gets `curl: (28) Connection timed out` (SYN swallowed), never refused.
+- `ss -ltn` shows the socket LISTENing; the process is fresh and healthy.
+- `sudo nft list ruleset | grep <port>` (or `iptables -L INPUT --line-numbers -v`) shows the catch-all `drop` rule ABOVE the specific ACCEPTs, with all counters on the drop and 0 on the accepts.
+
+## Root cause pattern
+`iptables -I CHAIN ...` inserts at position 1. A deployer block like:
+```
+-C/-I ACCEPT -s A ; -C/-I ACCEPT -s B ; -C/-I ACCEPT -s C ; -C/-I DROP
+```
+ends with DROP at position 1 — above every ACCEPT. Worse: `-C` checks existence, NEVER order, so re-running the block "verifies clean" while the chain stays poisoned.
+
+## Repair (converges on poisoned hosts)
+1. Delete the blanket DROP wherever it sits: `iptables -D INPUT ... -j DROP 2>/dev/null || true`
+2. Keep/insert the ACCEPTs at the head (`-C` before `-I`).
+3. APPEND the DROP (`-A`), so it is last.
+Delete-before-append keeps exactly one DROP per run (idempotent).
+
+## Behavioral test pattern (bats)
+Text greps cannot see order. Fake the kernel: write a small `iptables` shim to `$BATS_TEST_TMPDIR/bin` backed by an ordered file `$FW_STATE` (`-C`=grep -Fxq, `-I`=prepend, `-A`=append, `-D`=delete or exit 1), prepend it to `PATH`, `eval "$(sed -n '/^<install_fn>()/,/^}/p' script.sh)"`, stub `run_privileged(){ "$@"; }`, `systemctl(){ :; }`, `curl(){ return 0; }`, and **seed the state file with the poisoned order** before running. Assert: exactly one DROP, its line number equals `wc -l`, and every ACCEPT line number is smaller. Prove red by running the same scenario against `git show origin/main:<script>`.
+
+## Misc traps hit while doing this
+- zsh: `UID=...` is read-only/arith (renames blow up); a glob that matches nothing aborts the ENTIRE `&&`-less command line, silently skipping earlier `rm`s on that line.
+- `echo ===` in a remote zsh command parses as `=cmd` expansion — quote sentinel strings.
+- OpenWebUI `create_token` needs cwd `/app/backend` AND `WEBUI_SECRET_KEY` from PID 1's `/proc/1/environ` (`podman exec` sessions don't inherit the app process env).