Skip to content

feat: ship kafka-probe and network diagnostics in the runner image - #391

Closed
velo wants to merge 1 commit into
mainfrom
feat/image-network-diagnostics
Closed

feat: ship kafka-probe and network diagnostics in the runner image#391
velo wants to merge 1 commit into
mainfrom
feat/image-network-diagnostics

Conversation

@velo

@velo velo commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Why

A production pipeline failed with nothing but this, repeated forever:

org.apache.flink.util.FlinkRuntimeException: Failed to list subscribed topic partitions
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException:
    Timed out waiting for a node assignment. Call: listNodes

That single line is the enumerator's response to every pre-metadata failure. An empty bootstrap list, a DNS
failure, a dropped packet, a TLS mismatch and a rejected password are all indistinguishable in it. Triaging one
incident took a round trip to the customer to hand-patch log4j, redeploy, and mail back DEBUG logs — and the
image had no dig, no nc, no ss, so nothing could be checked from the pod either.

What

kafka-probe — splits the failure into DNS / TCP / TLS+SASL+metadata and names the layer that broke. It
builds an AdminClient from the connector classes already in /opt/flink/lib, so it exercises the same client
the job does rather than an approximation, and it defaults to the pod's own KAFKA_BOOTSTRAP_SERVERS and
SQRL_KAFKA_* variables, so it tests the real configuration instead of a retyped copy.

$ kubectl exec -it <taskmanager-pod> -- kafka-probe

== stage 1: DNS ==
  OK   b-1.example.kafka.us-east-1.amazonaws.com -> 10.229.33.220

== stage 2: TCP ==
  FAIL 10.229.33.220:9096 timed out after 5030 ms - packets dropped, typically a security group or NACL

VERDICT: TCP - hostnames resolve but no broker port accepts a connection - check security groups, NACLs,
         routing and VPC peering

KAFKA_PROBE_DEBUG=1 adds the NetworkClient and SASL handshake logs on stderr, so the DEBUG capture that
previously required editing cluster log4j and redeploying is now one env var. Client logs go to stderr and the
staged output to stdout, so the verdict stays greppable.

Diagnostic toolsdnsutils, netcat-openbsd, iproute2, unzip added to the existing apt-get layer.

Verification

Built the real image from flink-sql-runner/target/Dockerfile and ran every branch as the unprivileged flink
user, against a live Redpanda broker for the paths that need one:

Scenario Verdict produced
Empty bootstrap.servers FAIL [config] bootstrap.servers is empty
Unresolvable hostname DNS
Resolves, packets dropped TCP
TCP open, plaintext port SSLException
Broker up, topic absent UnknownTopicOrPartitionException
Broker up, topic present REACHABLE (6 partitions)

Worth noting the first two cases are exactly why the original error is so misleading: an empty or unresolvable
bootstrap list throws ConfigException at construction and never reaches listNodes, so that timeout actually
proves DNS resolved and narrows the problem to the network — the probe now says so directly.

Also confirmed Maven resource filtering (src/main/docker is copied with filtering=true) leaves the
${sys:...} and ${JAVA_HOME:-...} expressions byte-identical, mvn clean test is green, and
license:check / spotless:check pass.

Cost

~3 MB on a ~2.9 GB image:

dnsutils 34 KB | iproute2 3033 KB | netcat-openbsd 106 KB | unzip 375 KB

These are debugging tools in a runtime image, which is a real if small trade — async-profiler already set that
precedent here, and the alternative is that the next connectivity incident costs another customer round trip.
Happy to move them behind a build arg if you'd rather keep the default image lean.

Signed-off-by: Marvin Froeder <marvin@datasqrl.com>
@velo velo closed this Sep 2, 2026
@velo
velo deleted the feat/image-network-diagnostics branch September 2, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant