Hold HPKE key and context buffers out of line

[WillChilds-Klein]

Description

• EVP_HPKE_KEY is a public type the header invites callers to stack-allocate, so its size is ABI.
• Add ML-KEM support to HPKE (draft-ietf-hpke-pq-05) #3277 added the ML-KEM KEMs and grew it from 72 to 4808 bytes. A caller built against an earlier header reserves the smaller object, and the first HPKE call memsets the larger size over it.
• Its key material now lives in one allocation sized for the configured KEM, reached through a pointer. The struct is 32 bytes and does not move when a KEM is added.
• A static assert pins the size at four pointers, failing the build if KEM-dependent material moves back inline.
• EVP_HPKE_CTX is left alone. Its layout is identical on FIPS 5.x and main, so reshaping it would add an ABI break rather than repair one.

Call-outs

• This changes the layout of one public struct, so the four ABI Monitoring jobs are expected to be red. Whether that warrants an ABI_VERSION bump is a release-level decision I have not made here. The same question was raised in Add ML-KEM support to HPKE (draft-ietf-hpke-pq-05) #3277 and left open.
• Timing matters. HQC is the likely next KEM to reach HPKE, and its keys are much larger again than ML-KEM's: roughly 7.2 KB for an HQC-256 public key against 1568 bytes for ML-KEM-1024, which is the constant that sized the inline public_key field. Landing a KEM that large while the key material is still inline grows EVP_HPKE_KEY again, and the overflow with it.
• The 4808-byte layout exists only on main. No release branch or FIPS snapshot carries it, so no shipped artifact is affected.
• Keeping the type stack-allocatable is why the key material moves out of line rather than the struct becoming opaque. Hiding it would delete ScopedEVP_HPKE_KEY, force libssl's ECH config onto the heap, and diverge from upstream.

Testing / verification

Sizes, measured with sizeof against each tree's own headers:

Tree	EVP_HPKE_KEY	EVP_HPKE_CTX
FIPS 5.x (fips-2026-06-26-snapshot)	72	712
main today	4808	712
main + this PR	32	712

What a caller built against the FIPS 5.x header does when the library underneath it changes, which is the upgrade-without-rebuild case. The caller uses the public API only, and reserves the 5.x sizes:

5.x-built caller runs against	Bytes written past the reserved object	Bare stack object	Heap malloc(sizeof)
5.x libcrypto (control)	0	completes	completes
main, without this PR	4736	crashes	glibc heap abort
main, with this PR	0	completes	completes

Every case that does not crash completes a correct HPKE round trip. Plain X25519 is enough to trigger the overflow, because public_key moved to the offset one byte past the old object's end.

abidiff, same flags the ABI Monitoring jobs use:

Comparison	exit	symbols removed	type changed
FIPS 5.x -> main today (i.e. #3277)	4	0	evp_hpke_key_st 72 -> 4808
FIPS 5.x -> main + this PR	4	0	evp_hpke_key_st 72 -> 32
main today -> main + this PR	4	0	evp_hpke_key_st 4808 -> 32

• New tests cover what the indirection introduces: re-initializing a key across KEMs in both size directions, and reusing one key across two independent sender and recipient pairs.
• HPKE and ECH tests run under ASan, UBSan and leak detection.
• Reverting the key material to an inline array fails the build on evp_hpke_key_must_not_hold_inline_buffers, confirming the guard bites.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[github-actions]

🔒 Security Review — View Report

Please review before merging.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.64286% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.69%. Comparing base (b2fa449) to head (17ef5f9).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crypto/hpke/hpke.c	90.32%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3637      +/-   ##
==========================================
+ Coverage   78.46%   78.69%   +0.22%     
==========================================
  Files         705      705              
  Lines      130645   130738      +93     
  Branches    17912    17925      +13     
==========================================
+ Hits       102516   102889     +373     
+ Misses      27150    26869     -281     
- Partials      979      980       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.