Skip to content

finch kill removes a running container's port-forwarding rules, so published ports hang #1795

Description

@bnusunny

Description

finch kill removes a container's CNI hostport DNAT rules, even though the container keeps
running. After sending any signal with finch kill — including one the containerized
process ignores — published ports stop working for the lifetime of that container.

The host-side listener stays bound and still completes a TCP handshake, so clients do not
get a connection refused. They connect and then hang, because nothing DNATs the traffic
into the container any more. That makes it look like the containerized application has
stopped responding.

I hit this using finch kill -s HUP to ask a long-running process to reload its
configuration — a case where the container is deliberately meant to survive the signal.

Steps to reproduce

$ finch run -d --name nginxrepro -p 8899:80 public.ecr.aws/nginx/nginx:latest

$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://127.0.0.1:8899/
HTTP 200

$ finch kill -s USR1 nginxrepro          # a signal nginx does not exit on

$ finch ps --format '{{.Names}} {{.Status}}'
nginxrepro Up                             # container is still running

$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' --max-time 6 http://127.0.0.1:8899/
HTTP 000                                  # hangs until the timeout, no refusal

Expected behaviour

finch kill delivers a signal. If the container survives it, its published ports should
keep working. Only actual container termination should tear down the port forwarding.

Actual behaviour

The CNI-DN-* chain for the container is emptied. Before:

-A CNI-DN-765ef71cf868908533b60 -s 10.4.0.0/24 -p tcp -m tcp --dport 9997 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-765ef71cf868908533b60 -s 127.0.0.1/32 -p tcp -m tcp --dport 9997 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-765ef71cf868908533b60 -p tcp -m tcp --dport 9997 -j DNAT --to-destination 10.4.0.248:9997
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"bridge\" id: \"finch-39d46c6b…\"" -m multiport --dports 9997 -j CNI-DN-765ef71cf868908533b60

After finch kill -s USR1:

(no rules for 9997)

CNI-HOSTPORT-DNAT itself still exists and is still referenced from PREROUTING and
OUTPUT — only the per-container chain contents are gone. finch port still reports the
mapping (9997/tcp -> 0.0.0.0:9997), and inside the VM the pause process still holds the
listening socket, which is why connections are accepted rather than refused:

LISTEN     Recv-Q 2    0.0.0.0:9997    users:(("sleep",pid=…))   # queued, never accepted
ESTAB      Recv-Q 22 127.0.0.1:9997                             # request bytes, unread

Reachability from inside the container, and from the VM to the container's own IP, both
keep working throughout — only the host path via the published port breaks.

Notes

  • Not signal-specific. Reproduced with -s USR1 and -s HUP. A process that installs
    a no-op handler for the signal still loses its port forwarding, so this is not the
    application reacting to the signal.
  • finch stop followed by finch start restores the rules; there appears to be no way to
    re-establish them for a running container (finch update covers only cgroup resources).
  • Unaffected: connections from inside the container, and from the VM to <container-ip>:port.

Environment

finch version v1.17.2
Server Version: v2.2.1        (nerdctl)
limactl version 2.1.3-1-gf9eb6252
Kernel Version: 6.18.35-68.127.amzn2023.aarch64
Operating System: Amazon Linux 2023.12.20260622
Architecture: aarch64
Host: macOS 26.5.2 (25F84), arm64, vmType vz

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions