The dankclaw drain aborted at the alertmanager gate twice (2026-10-21): curl http://192.168.0.143:18093/health timed out beside a LISTENing socket. nft list ruleset showed the cause — in table ip filter chain INPUT the blanket tcp dport 18093 drop sat at position 1 with 159 packets, and all three ACCEPTs (loopback / host / bridge subnet) were dead code below it with 0 packets.
Root cause
install_signal_relay in deploy-remote.sh used -I INPUT for all four rules; -I prepends, so the DROP — inserted last — landed above every ACCEPT. Every probe to the relay was silently dropped, host-local included; Signal alert delivery could never have worked since that install order first ran.
Fix
Rebuild the tail instead of inserting it: delete any blanket DROP wherever it sits, keep the ACCEPTs (-C/-I at the head, unchanged), append the DROP. Converges on hosts poisoned by the old order; the live host was repaired by hand the same way this PR automates.
New bats case runs install_signal_relayfor real against an iptables sandbox seeded with the poisoned order: old code leaves DROP at position 1 (red), this code ends with ACCEPTs→DROP (green). -C existence greps could never see this bug — only order decides.
Observability page's Exposure section now carries the law.
Verification
red/green proof run against git show origin/main and the fix side by side
bats tests/observability_quadlet.bats → 47/47 ok
live host: /health answers 200 in 1.2 ms after the same reorder
## What broke
The dankclaw drain aborted at the alertmanager gate twice (2026-10-21): `curl http://192.168.0.143:18093/health` timed out beside a LISTENing socket. `nft list ruleset` showed the cause — in `table ip filter chain INPUT` the blanket `tcp dport 18093 drop` sat at position 1 with **159 packets**, and all three ACCEPTs (loopback / host / bridge subnet) were dead code below it with **0 packets**.
## Root cause
`install_signal_relay` in `deploy-remote.sh` used `-I INPUT` for all four rules; `-I` prepends, so the DROP — inserted last — landed **above** every ACCEPT. Every probe to the relay was silently dropped, host-local included; Signal alert delivery could never have worked since that install order first ran.
## Fix
- Rebuild the tail instead of inserting it: delete any blanket DROP wherever it sits, keep the ACCEPTs (`-C`/`-I` at the head, unchanged), **append** the DROP. Converges on hosts poisoned by the old order; the live host was repaired by hand the same way this PR automates.
- New bats case runs `install_signal_relay` **for real** against an `iptables` sandbox seeded with the poisoned order: old code leaves DROP at position 1 (red), this code ends with ACCEPTs→DROP (green). `-C` existence greps could never see this bug — only order decides.
- Observability page's Exposure section now carries the law.
## Verification
- red/green proof run against `git show origin/main` and the fix side by side
- `bats tests/observability_quadlet.bats` → 47/47 ok
- live host: `/health` answers 200 in 1.2 ms after the same reorder
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15d93-efb8-7108-b94d-4fabb14597ef (fix worktree: ~/.omp/wt/wt-fw-order)
iptables -I prepends, so the install block's final -I of the blanket DROP
landed it at position 1, above all three ACCEPTs: every packet to
18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each
ACCEPT). The host's own health probe timed out beside a LISTENing socket
and the whole dankclaw drain aborted at the alertmanager gate twice.
Install now rebuilds the tail: delete any blanket DROP wherever it sits,
keep the ACCEPTs at the head, append the DROP last - converges on hosts
poisoned by the old order. New bats case runs install_signal_relay for
real against an iptables sandbox seeded with the poisoned order (red on
the old code, green on this one).
dank
merged commit 0ecfe830f7 into main2026-10-21 16:59:16 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What broke
The dankclaw drain aborted at the alertmanager gate twice (2026-10-21):
curl http://192.168.0.143:18093/healthtimed out beside a LISTENing socket.nft list rulesetshowed the cause — intable ip filter chain INPUTthe blankettcp dport 18093 dropsat at position 1 with 159 packets, and all three ACCEPTs (loopback / host / bridge subnet) were dead code below it with 0 packets.Root cause
install_signal_relayindeploy-remote.shused-I INPUTfor all four rules;-Iprepends, so the DROP — inserted last — landed above every ACCEPT. Every probe to the relay was silently dropped, host-local included; Signal alert delivery could never have worked since that install order first ran.Fix
-C/-Iat the head, unchanged), append the DROP. Converges on hosts poisoned by the old order; the live host was repaired by hand the same way this PR automates.install_signal_relayfor real against aniptablessandbox seeded with the poisoned order: old code leaves DROP at position 1 (red), this code ends with ACCEPTs→DROP (green).-Cexistence greps could never see this bug — only order decides.Verification
git show origin/mainand the fix side by sidebats tests/observability_quadlet.bats→ 47/47 ok/healthanswers 200 in 1.2 ms after the same reordercwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15d93-efb8-7108-b94d-4fabb14597ef (fix worktree: ~/.omp/wt/wt-fw-order)