fix(stream): support upstream client certificate (mTLS) in L4 proxy#13596
Draft
AlinsRan wants to merge 7 commits into
Draft
fix(stream): support upstream client certificate (mTLS) in L4 proxy#13596AlinsRan wants to merge 7 commits into
AlinsRan wants to merge 7 commits into
Conversation
The stream (L4) subsystem could not present a client certificate when proxying to a TLS upstream, unlike the http subsystem. The http path injects the client cert via the apisix-nginx-module C API (set_cert_and_key), which has no stream counterpart. Instead, wire the native nginx stream proxy_ssl_certificate / proxy_ssl_certificate_key directives with variables, filled in the preread phase with the upstream tls.client_cert/client_key (or the ssl object referenced by tls.client_cert_id) using the inline data: PEM scheme. An empty value means no client certificate is presented. Fixes apache#12472
The stream upstream-mtls test referenced the upstream server certs via `../t/certs/...`, which resolves to `t/servroot/conf/../t/certs/...` (a non-existent path) and made nginx fail to start. Use the same `../../certs/...` prefix as other .t tests (e.g. healthcheck-https.t), which resolves to the repo `t/certs/` directory. The new `proxy_ssl_certificate` directive in every stream server block made the per-port PROXY protocol cli test mis-match: its plain-vs-TLS `grep -E "ssl_certificate "` matched the substring inside `proxy_ssl_certificate `. Anchor the grep on a leading whitespace boundary so it only matches the downstream `ssl_certificate` directive.
The stream mTLS path fed `tls.client_key` (or the ssl object key) directly into the `data:` proxy_ssl_certificate_key variable. That key is stored AES-encrypted at rest, so nginx received base64 ciphertext instead of PEM and failed with "cannot load certificate key". Decrypt it with aes_decrypt_pkey (a no-op for plaintext PEM) before building the data: value, matching the http path which decrypts via fetch_pkey. Also correct the no-client-cert test assertion: an mTLS upstream that rejects a certless handshake logs "client sent no required SSL certificate", not "upstream SSL certificate verify error" (which only applies to proxy_ssl_verify server-cert checks).
Replace the interim proxy_ssl_certificate data: plumbing with the new stream set_cert_and_key C-API shipped in APISIX-Runtime 1.3.8 (apisix-nginx-module 1.19.6). The cert/key are parsed and cached once and applied at the upstream SSL handshake, so the decrypted private key is never stringified into an nginx variable. Bump APISIX_RUNTIME 1.3.6 -> 1.3.8.
- init.lua: log only client ssl id/type instead of delay-encoding the whole ssl object, which could emit certificate/key material at info level. - upstream.lua: guard set_tls and set_cert_and_key independently so an older runtime exposing the stream module without set_cert_and_key falls back gracefully instead of calling a nil value.
Add repeat_each(1)/no_long_string()/no_shuffle()/no_root_location() per the test harness convention; no_shuffle keeps the setup-then-hit ordering stable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #12472
The stream (L4 TCP/TLS) subsystem could not present a client certificate (mTLS) when APISIX proxies to a TLS upstream, unlike the http subsystem which honors upstream
tls.client_cert/client_key/client_cert_id.apisix/upstream.luaset the stream upstream TLS (SNI/enable) but never applied the client certificate.Approach
The http subsystem injects the client cert per-request through the apisix-nginx-module C API
ngx_http_apisix_upstream_set_cert_and_key. The stream subsystem now has the equivalentngx_stream_apisix_upstream_set_cert_and_key, shipped in APISIX-Runtime 1.3.8 (apisix-nginx-module 1.19.6, api7/apisix-nginx-module#114). This PR uses it so the stream path mirrors http exactly:apisix/upstream.lua: in the streamscheme == "tls"branch, resolve the cert/key fromup_conf.tls.client_cert/client_key(inline) or from the ssl object referenced bytls.client_cert_id, parse + decrypt them once viaapisix_ssl.fetch_cert/fetch_pkey(cached), and apply them withresty.apisix.stream.upstream.set_cert_and_key. The decrypted private key is held only as an opaque parsed object and never stringified into an nginx variable.apisix/init.lua: extract theclient_cert_id->api_ctx.upstream_sslresolution into a sharedresolve_upstream_client_certhelper, called fromstream_preread_phasetoo (previously http-only)..requirements/ci/linux-install-openresty.sh: bumpAPISIX_RUNTIME1.3.6 -> 1.3.8 (and the runtime-debug.debchecksums).t/stream-node/upstream-mtls.t— real mTLS handshake against assl_verify_client onupstream: success with inline cert and withclient_cert_id, and rejection when no client cert is presented.docs/en/latest/mtls.md.No schema change needed —
tls.client_cert/client_key/client_cert_idalready exist on the upstream schema.