Repository navigation
feat(transport): send asynchronously with prim_inet:send_async/3 - #236
Merged
zmstone merged 4 commits intoOct 23, 2026
Merged
Conversation
Since OTP 26 the inet driver requires a caller tag in front of the
data, so async_send/3 became gen_tcp:send/2 plus a reply message that
the caller sends to itself. Each send then waits for the driver.
EMQX OTP builds add gen_tcp:send_async/3. It sends with a constant tag
and returns when the driver has the data. The driver replies later
with {inet_reply, Sock, Status, send_async}.
async_send/3 now uses gen_tcp:send_async/3 for a TCP port when the
running OTP exports it. The check runs once and its result is kept in a
persistent term. With nosuspend and a busy port it returns
{error, busy} and queues nothing, as before OTP 26.
On OTP 26 and later, the fallback and the TLS path now send the same
four-element reply, so a caller handles one shape. This changes the
reply from {inet_reply, Sock, ok} to {inet_reply, Sock, ok, send_async}.
OTP 25 and older keep the three-element reply of the driver.
Merged
3 tasks done
zmstone
added a commit
to zmstone/emqx
that referenced
this pull request
Oct 18, 2026
…eply gen_tcp:send/2 gives the data to the inet driver and then waits for the driver's reply message, so every outgoing packet costs the connection process a round trip through the port. EMQX 4.4 used erlang:port_command/3 through esockd_transport:async_send/3 and took the reply from its mailbox later. That path stopped working on OTP 26, where the driver requires a caller tag in front of the data, and esockd made async_send/3 synchronous. The EMQX OTP fork adds gen_tcp:send_async/3 for the inet backend (emqx/otp#122): it sends with a constant tag and returns when the driver has the data. The reply comes later as {inet_reply, Socket, Status, send_async}, one per accepted send, in send order; a dead port sends no reply and the owner sees the port exit as before. esockd_transport:async_send/3 uses it when the running OTP exports it (emqx/esockd#236). The connection uses esockd_transport:async_send/3 with nosuspend when the socket is an inet port and the running OTP exports gen_tcp:send_async/3. On {error, busy} (driver queue above its high watermark) the packet is sent synchronously, which keeps today's back-pressure. A reply with an error is handled as a synchronous send error. The reply of an older esockd ({inet_reply, Socket, Status}) is accepted too. Other transports and OTP builds without the function keep the synchronous send.
zmstone
added a commit
to zmstone/emqx
that referenced
this pull request
Oct 18, 2026
…eply gen_tcp:send/2 gives the data to the inet driver and then waits for the driver's reply message, so every outgoing packet costs the connection process a round trip through the port. EMQX 4.4 used erlang:port_command/3 through esockd_transport:async_send/3 and took the reply from its mailbox later. That path stopped working on OTP 26, where the driver requires a caller tag in front of the data, and esockd made async_send/3 synchronous. The EMQX OTP fork adds gen_tcp:send_async/3 for the inet backend (emqx/otp#122): it sends with a constant tag and returns when the driver has the data. The reply comes later as {inet_reply, Socket, Status, send_async}, one per accepted send, in send order; a dead port sends no reply and the owner sees the port exit as before. esockd_transport:async_send/3 uses it when the running OTP exports it (emqx/esockd#236). The connection uses esockd_transport:async_send/3 with nosuspend when the socket is a port (gen_tcp, inet backend). EMQX runs only on the EMQX OTP fork, so the connection does not check that gen_tcp:send_async/3 exists. On {error, busy} (driver queue above its high watermark) the packet is sent synchronously, which keeps the back-pressure. A reply with an error is handled as a synchronous send error. The reply of an older esockd ({inet_reply, Socket, Status}) is accepted too. TLS and QUIC sockets keep the synchronous send.
The EMQX OTP fork moved send_async/3 from gen_tcp to prim_inet during review: the function works only on inet driver ports and takes port_command options. Probe prim_inet instead of gen_tcp.
zmstone
added a commit
to zmstone/emqx
that referenced
this pull request
Oct 21, 2026
…eply gen_tcp:send/2 gives the data to the inet driver and then waits for the driver's reply message, so every outgoing packet costs the connection process a round trip through the port. EMQX 4.4 used erlang:port_command/3 through esockd_transport:async_send/3 and took the reply from its mailbox later. That path stopped working on OTP 26, where the driver requires a caller tag in front of the data, and esockd made async_send/3 synchronous. The EMQX OTP fork adds gen_tcp:send_async/3 for the inet backend (emqx/otp#122): it sends with a constant tag and returns when the driver has the data. The reply comes later as {inet_reply, Socket, Status, send_async}, one per accepted send, in send order; a dead port sends no reply and the owner sees the port exit as before. esockd_transport:async_send/3 uses it when the running OTP exports it (emqx/esockd#236). The connection uses esockd_transport:async_send/3 with nosuspend when the socket is a port (gen_tcp, inet backend). EMQX runs only on the EMQX OTP fork, so the connection does not check that gen_tcp:send_async/3 exists. On {error, busy} (driver queue above its high watermark) the packet is sent synchronously, which keeps the back-pressure. A reply with an error is handled as a synchronous send error. The reply of an older esockd ({inet_reply, Socket, Status}) is accepted too. TLS and QUIC sockets keep the synchronous send.
zmstone
changed the title
The image has prim_inet:send_async/3, so the asynchronous send test runs instead of being skipped.
zmstone
commented
Oct 23, 2026
zmstone
commented
Oct 23, 2026
zmstone
marked this pull request as ready for review
October 23, 2026 11:45
thalesmg
approved these changes
Oct 23, 2026
zmstone
added a commit
to zmstone/emqx
that referenced
this pull request
Oct 26, 2026
…eply gen_tcp:send/2 gives the data to the inet driver and then waits for the driver's reply message, so every outgoing packet costs the connection process a round trip through the port. EMQX 4.4 used erlang:port_command/3 through esockd_transport:async_send/3 and took the reply from its mailbox later. That path stopped working on OTP 26, where the driver requires a caller tag in front of the data, and esockd made async_send/3 synchronous. The EMQX OTP fork adds gen_tcp:send_async/3 for the inet backend (emqx/otp#122): it sends with a constant tag and returns when the driver has the data. The reply comes later as {inet_reply, Socket, Status, send_async}, one per accepted send, in send order; a dead port sends no reply and the owner sees the port exit as before. esockd_transport:async_send/3 uses it when the running OTP exports it (emqx/esockd#236). The connection uses esockd_transport:async_send/3 with nosuspend when the socket is a port (gen_tcp, inet backend). EMQX runs only on the EMQX OTP fork, so the connection does not check that gen_tcp:send_async/3 exists. On {error, busy} (driver queue above its high watermark) the packet is sent synchronously, which keeps the back-pressure. A reply with an error is handled as a synchronous send error. The reply of an older esockd ({inet_reply, Socket, Status}) is accepted too. TLS and QUIC sockets keep the synchronous send.
zmstone
added a commit
to zmstone/emqx
that referenced
this pull request
Oct 28, 2026
…eply gen_tcp:send/2 gives the data to the inet driver and then waits for the driver's reply message, so every outgoing packet costs the connection process a round trip through the port. EMQX 4.4 used erlang:port_command/3 through esockd_transport:async_send/3 and took the reply from its mailbox later. That path stopped working on OTP 26, where the driver requires a caller tag in front of the data, and esockd made async_send/3 synchronous. The EMQX OTP fork adds gen_tcp:send_async/3 for the inet backend (emqx/otp#122): it sends with a constant tag and returns when the driver has the data. The reply comes later as {inet_reply, Socket, Status, send_async}, one per accepted send, in send order; a dead port sends no reply and the owner sees the port exit as before. esockd_transport:async_send/3 uses it when the running OTP exports it (emqx/esockd#236). The connection uses esockd_transport:async_send/3 with nosuspend when the socket is a port (gen_tcp, inet backend). EMQX runs only on the EMQX OTP fork, so the connection does not check that gen_tcp:send_async/3 exists. On {error, busy} (driver queue above its high watermark) the packet is sent synchronously, which keeps the back-pressure. A reply with an error is handled as a synchronous send error. The reply of an older esockd ({inet_reply, Socket, Status}) is accepted too. TLS and QUIC sockets keep the synchronous send.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
esockd_transport:async_send/3sends asynchronously again on OTP 26 and later when the running OTP exportsprim_inet:send_async/3.Why
Since OTP 26 (OTP-18520), the inet driver requires a caller tag in front of the data. A bare
erlang:port_command/3fails withEINVAL. Soasync_send/3becamegen_tcp:send/2plus a reply message that the caller sends to itself (3b304e2). Each send waits for the driver.The EMQX OTP fork adds
prim_inet:send_async/2,3(emqx/otp#122). It sends with a constant tag and returns when the driver has the data. The driver replies later with{inet_reply, Sock, Status, send_async}, one reply perok, in send order. In an EMQX benchmark (QoS 0, 1:1 publish/subscribe, 20k messages/s, 40k connections), this lowered broker CPU per message by 12%.Change
async_send/3callsprim_inet:send_async/3when the running OTP exports it. The check runs once. The result is kept in the persistent term{esockd_transport, send_async_module}.nosuspendand a busy port, the function returns{error, busy}and queues nothing. This is the behavior before OTP 26. The caller decides what to do, for example send again withsend/2.gen_tcp:send/2, as before.Reply shape (breaking)
On OTP 26 and later, every path now sends the same reply:
{inet_reply, Sock, ok | {error, Reason}, send_async}. The fallback and the TLS path sent{inet_reply, Sock, ok}before. A caller that matches the three-element reply must also match the four-element reply. OTP 25 and older keep the three-element reply of the driver.The test echo servers and the examples now match both shapes. No caller in EMQX
dev-63usesasync_send; the EMQX change that uses it handles both shapes. Because of this change, the next tag can be a minor version bump.Tests
t_async_send_reply: the reply has four elements and the peer receives the data.t_async_send_fallback: with the persistent term set toundefined, the send is synchronous and the reply is sent by the function; a closed socket returns{error, closed}.t_async_send_prim_inet_send_async: skipped whenprim_inet:send_async/3is not exported. Otherwise:{error, busy}withnosuspendon a full port, and oneokreply per accepted send after the peer reads the data.Results:
make xrefandrebar3 dialyzerclean;rebar3 as test ct242 passed, 4 skipped (the new asynchronous case, andesockd_socket_SUITEwhich needs the compose test server).6.2-4image):esockd_transport_SUITE36 passed, the asynchronous case included. CI runs therun_test_casejob on this image too.esockd_transport.erland the suite compile.Release
The asynchronous path needs EMQX OTP
28.4.1-9or later (emqx/otp#122); on other OTP builds the behavior is the fallback described above. emqx/emqx#19627 uses this function and bumps the esockd tag inmix.exsafter the tag exists.