33 lines
2.4 KiB
Markdown
33 lines
2.4 KiB
Markdown
---
|
|
name: iptables-insert-order-drop-shadows-accepts
|
|
description: "Diagnose or prevent a service port that times out beside a LISTENing socket because a deployer's iptables -I sequence landed a blanket DROP above its ACCEPTs; includes the sandboxed fake-iptables red/green bats pattern."
|
|
---
|
|
|
|
# iptables `-I` order trap: blanket DROP shadowing ACCEPTs
|
|
|
|
## Failure signature
|
|
- Client to a port gets `curl: (28) Connection timed out` (SYN swallowed), never refused.
|
|
- `ss -ltn` shows the socket LISTENing; the process is fresh and healthy.
|
|
- `sudo nft list ruleset | grep <port>` (or `iptables -L INPUT --line-numbers -v`) shows the catch-all `drop` rule ABOVE the specific ACCEPTs, with all counters on the drop and 0 on the accepts.
|
|
|
|
## Root cause pattern
|
|
`iptables -I CHAIN ...` inserts at position 1. A deployer block like:
|
|
```
|
|
-C/-I ACCEPT -s A ; -C/-I ACCEPT -s B ; -C/-I ACCEPT -s C ; -C/-I DROP
|
|
```
|
|
ends with DROP at position 1 — above every ACCEPT. Worse: `-C` checks existence, NEVER order, so re-running the block "verifies clean" while the chain stays poisoned.
|
|
|
|
## Repair (converges on poisoned hosts)
|
|
1. Delete the blanket DROP wherever it sits: `iptables -D INPUT ... -j DROP 2>/dev/null || true`
|
|
2. Keep/insert the ACCEPTs at the head (`-C` before `-I`).
|
|
3. APPEND the DROP (`-A`), so it is last.
|
|
Delete-before-append keeps exactly one DROP per run (idempotent).
|
|
|
|
## Behavioral test pattern (bats)
|
|
Text greps cannot see order. Fake the kernel: write a small `iptables` shim to `$BATS_TEST_TMPDIR/bin` backed by an ordered file `$FW_STATE` (`-C`=grep -Fxq, `-I`=prepend, `-A`=append, `-D`=delete or exit 1), prepend it to `PATH`, `eval "$(sed -n '/^<install_fn>()/,/^}/p' script.sh)"`, stub `run_privileged(){ "$@"; }`, `systemctl(){ :; }`, `curl(){ return 0; }`, and **seed the state file with the poisoned order** before running. Assert: exactly one DROP, its line number equals `wc -l`, and every ACCEPT line number is smaller. Prove red by running the same scenario against `git show origin/main:<script>`.
|
|
|
|
## Misc traps hit while doing this
|
|
- zsh: `UID=...` is read-only/arith (renames blow up); a glob that matches nothing aborts the ENTIRE `&&`-less command line, silently skipping earlier `rm`s on that line.
|
|
- `echo ===` in a remote zsh command parses as `=cmd` expansion — quote sentinel strings.
|
|
- OpenWebUI `create_token` needs cwd `/app/backend` AND `WEBUI_SECRET_KEY` from PID 1's `/proc/1/environ` (`podman exec` sessions don't inherit the app process env).
|