command-center/agent/managed-skills/iptables-insert-order-drop-shadows-accepts/SKILL.md

name, description

name	description
iptables-insert-order-drop-shadows-accepts	Diagnose or prevent a service port that times out beside a LISTENing socket because a deployer's iptables -I sequence landed a blanket DROP above its ACCEPTs; includes the sandboxed fake-iptables red/green bats pattern.

iptables -I order trap: blanket DROP shadowing ACCEPTs

Failure signature

• Client to a port gets curl: (28) Connection timed out (SYN swallowed), never refused.
• ss -ltn shows the socket LISTENing; the process is fresh and healthy.
• sudo nft list ruleset | grep <port> (or iptables -L INPUT --line-numbers -v) shows the catch-all drop rule ABOVE the specific ACCEPTs, with all counters on the drop and 0 on the accepts.

Root cause pattern

iptables -I CHAIN ... inserts at position 1. A deployer block like:

-C/-I ACCEPT -s A ; -C/-I ACCEPT -s B ; -C/-I ACCEPT -s C ; -C/-I DROP

ends with DROP at position 1 — above every ACCEPT. Worse: -C checks existence, NEVER order, so re-running the block "verifies clean" while the chain stays poisoned.

Repair (converges on poisoned hosts)

1. Delete the blanket DROP wherever it sits: iptables -D INPUT ... -j DROP 2>/dev/null || true
2. Keep/insert the ACCEPTs at the head (-C before -I).
3. APPEND the DROP (-A), so it is last. Delete-before-append keeps exactly one DROP per run (idempotent).

Behavioral test pattern (bats)

Text greps cannot see order. Fake the kernel: write a small iptables shim to $BATS_TEST_TMPDIR/bin backed by an ordered file $FW_STATE (-C=grep -Fxq, -I=prepend, -A=append, -D=delete or exit 1), prepend it to PATH, eval "$(sed -n '/^<install_fn>()/,/^}/p' script.sh)", stub run_privileged(){ "$@"; }, systemctl(){ :; }, curl(){ return 0; }, and seed the state file with the poisoned order before running. Assert: exactly one DROP, its line number equals wc -l, and every ACCEPT line number is smaller. Prove red by running the same scenario against git show origin/main:<script>.

Misc traps hit while doing this

• zsh: UID=... is read-only/arith (renames blow up); a glob that matches nothing aborts the ENTIRE &&-less command line, silently skipping earlier rms on that line.
• echo === in a remote zsh command parses as =cmd expansion — quote sentinel strings.
• OpenWebUI create_token needs cwd /app/backend AND WEBUI_SECRET_KEY from PID 1's /proc/1/environ (podman exec sessions don't inherit the app process env).