1
0
mirror of https://github.com/MariaDB/server.git synced 2026-10-23 18:49:29 +03:00
Files
mariadb/sql
Vladislav Vaintroub fb641b0cd1 MDEV-40287 AES_ENCRYPT() and KDF() return NULL with OpenSSL 4.0
MyCTX used an EVP_CIPHER_CTX in a stack buffer instead of allocating it
with EVP_CIPHER_CTX_new(). OpenSSL 4.0 rejects a context that was not
created that way for ciphers that use an IV: EVP_CipherInit_ex() fails
with "invalid iv length", so AES_ENCRYPT() and KDF() in CBC/CTR/GCM
modes return NULL. ECB has no IV and still works, which is why only the
non-ECB modes broke.

Allocate the context with EVP_CIPHER_CTX_new()/EVP_CIPHER_CTX_free(), and
on an init error free and null it, so a later update()/finish() from a
caller that ignored the error is a safe no-op, not a double free. Remove
check_openssl_compatibility() and the EVP_CIPHER_CTX_SIZE and
EVP_CIPHER_CTX_init macros.

my_md5() used the same stack-buffer-sized-by-EVP_MD_CTX_SIZE hack, whose
only guard was check_openssl_compatibility(). MD5_CTX is a concrete struct
in both OpenSSL and WolfSSL, so use it directly with MD5_Init/Update/Final
on a caller-owned context, like my_sha.inl does for SHA*_CTX; this removes
the EVP_MD_CTX hack and EVP_MD_CTX_SIZE. The exception is OpenSSL 1.0.x,
which aborts on the low-level MD5 API in FIPS mode (MDEV-7788): there, and
only there, keep the EVP path with the non-FIPS flag. 1.1.0 removed the
FIPS Object Module and 3.0+ low-level MD5 bypasses the provider, so the
concrete MD5_CTX is correct on every other version. SHA is unaffected, it
already used concrete SHA*_CTX structs.

Verified against OpenSSL 4.0.1: the mysys aes-t test fails on the
CBC/CTR/GCM cases with the stack buffer and passes with
EVP_CIPHER_CTX_new(). The 1.0.x MD5 path verified against a FIPS build of
OpenSSL 1.0.2u: the EVP path computes MD5, the low-level path aborts.

No visible performance degradation: the extra allocation costs
~12 ns/call (WolfSSL) and ~30 ns (OpenSSL) on Windows at a 30-byte
payload, nothing at 16 KB, and nothing on Linux/glibc; sysbench OLTP
over encrypted tables and redo log is unchanged.

Tests main.tlsv13 and main.ssl_crl: OpenSSL 4.0.3 reports TLS alerts as
"tls alert ..." (was "ssl/tls alert ..."), accept both.

Assisted-by: Claude:claude-opus-4-8
2026-10-20 01:59:37 +02:00
..
…
2026-07-23 11:12:27 +02:00
2026-07-23 11:12:27 +02:00
…
2025-09-04 18:08:39 +03:00
2024-01-17 13:35:05 +02:00
…
…
…
2024-11-29 13:43:17 +02:00
2024-11-29 12:37:46 +02:00
2024-10-29 15:24:38 +01:00
…
2025-09-04 18:08:39 +03:00
2023-12-17 11:20:43 +01:00
…
…
…
2025-02-27 04:02:33 +01:00
2025-09-04 18:08:39 +03:00
2024-11-29 13:43:17 +02:00
…
…
2024-11-29 13:43:17 +02:00
2024-12-05 10:11:58 +01:00
2026-08-19 08:13:54 +02:00
2026-01-28 14:22:20 +01:00
2026-01-28 14:22:20 +01:00
2026-01-28 14:22:20 +01:00
2026-01-28 14:22:20 +01:00
2026-07-23 11:12:27 +02:00
2026-01-28 14:22:20 +01:00
2024-11-29 13:43:17 +02:00
2023-11-08 16:23:30 +01:00
…
2024-06-27 10:26:09 +03:00
2024-06-24 13:09:47 +03:00
…
…
2025-09-04 18:08:39 +03:00
…
2023-11-08 16:23:30 +01:00
2023-10-14 13:36:11 +03:00
2026-01-28 14:22:20 +01:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2026-04-20 11:07:30 +03:00
2023-12-18 11:19:04 +01:00
2024-11-29 12:37:46 +02:00
…
2024-06-24 13:09:47 +03:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2024-07-22 15:14:50 +02:00
2026-06-15 10:43:23 +02:00
2026-06-15 10:43:23 +02:00
2025-10-28 17:49:51 +01:00
2025-10-28 17:49:51 +01:00
2026-07-23 11:12:27 +02:00
2024-11-29 12:37:46 +02:00
2026-08-19 08:13:54 +02:00
2025-09-04 18:08:39 +03:00
2025-01-10 13:14:42 +01:00
2024-06-24 13:09:47 +03:00
2025-01-30 11:55:13 +01:00
2025-01-30 11:55:13 +01:00
2023-07-10 11:22:21 +03:00
2024-11-29 13:43:17 +02:00
2024-11-29 13:43:17 +02:00
2026-08-19 08:13:54 +02:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2024-11-23 08:14:22 -07:00
2026-01-02 11:06:05 +02:00
2026-07-23 11:12:27 +02:00
2026-07-23 11:12:27 +02:00
2026-03-17 18:57:42 +01:00
2026-01-28 14:22:20 +01:00
…
2024-08-20 10:00:39 +02:00
2025-09-04 18:08:39 +03:00
2024-06-27 10:26:09 +03:00
2024-11-29 12:37:46 +02:00
…
2025-09-19 12:31:14 +03:00
2025-09-04 18:08:39 +03:00
2023-05-23 12:25:39 +03:00
2024-06-11 12:50:10 +03:00
2024-04-15 18:54:30 +02:00
2026-01-28 14:22:20 +01:00
2024-07-03 12:45:30 +02:00
2023-08-04 08:01:06 +02:00
2026-07-23 11:12:27 +02:00
2023-12-17 11:20:43 +01:00
2026-07-23 11:12:27 +02:00
2023-12-02 01:02:50 +01:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2026-03-17 11:05:34 +02:00
2024-03-27 15:00:56 +02:00
2025-09-04 18:08:39 +03:00
2026-07-23 11:12:27 +02:00
2025-01-20 09:57:37 +02:00
2025-03-27 08:01:47 +02:00
2023-09-25 13:06:57 +10:00
2024-01-18 19:22:23 +02:00
2023-08-01 15:08:52 +02:00
2024-11-23 08:14:22 -07:00
2024-11-23 08:14:23 -07:00
2026-08-19 08:13:54 +02:00
2026-07-23 11:12:27 +02:00
2026-07-23 11:12:27 +02:00
2024-01-10 18:01:46 +11:00
2025-09-04 18:08:39 +03:00
2025-09-04 18:08:39 +03:00
2024-06-24 13:09:47 +03:00
…
2025-09-04 18:08:39 +03:00
…
2024-11-29 13:43:17 +02:00
2024-01-02 17:37:58 +02:00
2024-06-27 10:26:09 +03:00
2026-06-21 15:32:20 +02:00
2024-01-03 15:37:57 +02:00
2024-10-03 10:55:08 +03:00
2026-07-23 11:12:27 +02:00
2026-07-23 11:12:27 +02:00
2026-06-21 15:32:20 +02:00