[zmstone]

Problem

A TLS connect storm can wedge the OTP ssl listener supervisor for hours.

ssl:handshake/3 starts the connection processes with
supervisor:start_child(tls_connection_sup, ...), and each start scans that
supervisor's whole mailbox, because proc_lib:sync_start/2 waits in a selective
receive the compiler cannot mark. The cost of one start grows with the backlog,
so a large enough backlog never clears. Callers wait in gen_server:call with
an infinity timeout, so nothing times out and nothing recovers.

Measured on a node that had taken 300,000 TLS connections:

Measurement	Value
tls_connection_sup mailbox (messages)	300,000, constant
Child starts completed (per second)	38.7
Supervisor reductions (per second)	12.0M
Surviving TLS connections	0

Reductions per start matched the mailbox length, so nearly all of the CPU was
mailbox scanning.

Change

Refuse the handshake while the supervisor mailbox, which is the number of
handshakes waiting to start, is at or above max_pending_handshakes. A short
backlog keeps each start cheap.

max_pending_handshakes is an esockd application environment value, declared in
esockd.app.src next to socket_fast_close_method and read the same way. It is
a node-wide setting rather than a per-listener option. The default is 100,
infinity disables the check, and 0 refuses everything, which the tests use.

The refusal reason is the bare atom too_many_handshakes, so
esockd_connection_sup counts it under its own shutdown key rather than
grouping it with real TLS errors.

The check returns false when tls_connection_sup is not registered, so a
DTLS-only listener and a node without the ssl application behave as before.

Choosing the default

Measured on a 20 core host with OTP 28.4.1 and TLS 1.2, 3,000 connections per
run. The server uses a fixed pool of 16 acceptors, each handing the accepted
socket to its own process for the handshake, which is how esockd drives it, so
in-flight server handshakes are not bounded by the acceptor count. Client and
server share the node, so each connection costs two start_child calls and the
mailbox holds requests from both sides.

Concurrent client connects	Acceptors	Rate (conns/s)	Peak mailbox (messages)	Time at mailbox >= 100 (%)
8	16	1,203	16	0.00
16	16	1,215	32	0.00
32	16	1,537	41	0.00
64	16	1,251	71	0.00
128	16	892	75	0.00
256	16	390	157	0.06

One run per row, so read the rate column as roughly 1,200 with noise.

Peak throughput of about 1,500 connections per second is 3,000 start_child
calls per second, so roughly 350 microseconds of supervisor time for one call.
The supervisor is a single process, which makes that a node-wide ceiling.

The queue stays below 100 across the whole range where throughput holds up. It
only reaches 100 at 256 concurrent connects, where the rate has already
collapsed to 390 per second because the host is saturated, and even then for
0.06 percent of the samples. So a limit of 100, about 35 ms of queued
supervisor work, does not cost throughput. It discards backlog that was not
going to raise the rate.

Scope

This targets 5.18.x only. Older lines are out of scope, because a listener that
keeps its connection rate limit reaches that limit before the supervisor
backlog can build.

Verification

On a 300,000 connection test with this change, the supervisor mailbox stayed at
0, 161,440 connections were established and 138,560 were refused.

make xref, make dialyzer, make eunit and make ct pass on OTP 28.4.1. CT
is 246 passed and 4 skipped, the skips being pre-existing environment ones. Four
new cases cover the declared default, infinity, the refusal, and an idle
supervisor.

[zmstone]

better option:
emqx/otp#129