MyCTX used an EVP_CIPHER_CTX in a stack buffer instead of allocating it
with EVP_CIPHER_CTX_new(). OpenSSL 4.0 rejects a context that was not
created that way for ciphers that use an IV: EVP_CipherInit_ex() fails
with "invalid iv length", so AES_ENCRYPT() and KDF() in CBC/CTR/GCM
modes return NULL. ECB has no IV and still works, which is why only the
non-ECB modes broke.
Allocate the context with EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free(), and
on an init error free and null it, so a later update()/finish() from a
caller that ignored the error is a safe no-op, not a double free. Remove
check_openssl_compatibility() and the EVP_CIPHER_CTX_SIZE and
EVP_CIPHER_CTX_init macros.
my_md5() used the same stack-buffer-sized-by-EVP_MD_CTX_SIZE hack, whose
only guard was check_openssl_compatibility(). MD5_CTX is a concrete struct
in both OpenSSL and WolfSSL, so use it directly with MD5_Init/Update/Final
on a caller-owned context, like my_sha.inl does for SHA*_CTX; this removes
the EVP_MD_CTX hack and EVP_MD_CTX_SIZE. The exception is OpenSSL 1.0.x,
which aborts on the low-level MD5 API in FIPS mode (MDEV-7788): there, and
only there, keep the EVP path with the non-FIPS flag. 1.1.0 removed the
FIPS Object Module and 3.0+ low-level MD5 bypasses the provider, so the
concrete MD5_CTX is correct on every other version. SHA is unaffected, it
already used concrete SHA*_CTX structs.
Verified against OpenSSL 4.0.1: the mysys aes-t test fails on the
CBC/CTR/GCM cases with the stack buffer and passes with
EVP_CIPHER_CTX_new(). The 1.0.x MD5 path verified against a FIPS build of
OpenSSL 1.0.2u: the EVP path computes MD5, the low-level path aborts.
No visible performance degradation: the extra allocation costs
~12 ns/call (WolfSSL) and ~30 ns (OpenSSL) on Windows at a 30-byte
payload, nothing at 16 KB, and nothing on Linux/glibc; sysbench OLTP
over encrypted tables and redo log is unchanged.
Tests main.tlsv13 and main.ssl_crl: OpenSSL 4.0.3 reports TLS alerts as
"tls alert ..." (was "ssl/tls alert ..."), accept both.
Assisted-by: Claude:claude-opus-4-8
fb641b0cd1