mirror of
https://github.com/keycloak/keycloak.git
synced 2026-10-23 19:29:23 +03:00
Check the Unicode setup of the database on startup
Closes #46557 Signed-off-by: Pedro Ruivo <1492066+pruivo@users.noreply.github.com> Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com> Co-authored-by: Pedro Ruivo <1492066+pruivo@users.noreply.github.com> Co-authored-by: Alexander Schwartz <alexander.schwartz@ibm.com>
This commit is contained in:
Pedro Ruivo
GitHub
@@ -169,6 +169,19 @@ The default value is 5 minutes, which is the same value used by the current {pro
|
||||
* `transaction-migration-timeout`: Controls the timeout for transactions responsible for migrating the database schema to a new version.
|
||||
The default value is 30 minutes, as schema migrations may take a while to complete for large tables.
|
||||
|
||||
=== Database character encoding checks during startup
|
||||
|
||||
{project_name} will log a warning if the database is not configured to use UTF-8 encoding.
|
||||
This check applies to all supported databases.
|
||||
|
||||
For details on configuring UTF-8 encoding for your database, see the https://www.keycloak.org/server/db[Configuring the database] guide.
|
||||
|
||||
As part of this change, the database configuration documentation has been updated:
|
||||
|
||||
* For MySQL and MariaDB, the documentation now recommends using `utf8mb4` as the character set.
|
||||
The previously documented `utf8mb3` (or `utf8`) character set has been removed from the documentation due to its limitations in storing certain Unicode characters.
|
||||
* Previously recommended JDBC driver settings for Oracle, MySQL, and MariaDB have been removed from the documentation, as current versions of these databases use appropriate default values.
|
||||
|
||||
// ------------------------ Deprecated features ------------------------ //
|
||||
== Deprecated features
|
||||
|
||||
@@ -194,6 +207,12 @@ The methods `getUserSessionsStream(RealmModel realm, ClientModel client)`, `getU
|
||||
They are replaced with read-only alternatives to reduce memory consumption.
|
||||
See the `UserSessionProvider` API changes section in notable changes for further information.
|
||||
|
||||
=== Non-UTF-8 database character encoding deprecated
|
||||
|
||||
Running {project_name} on a database that is not using UTF-8 as a charset is now deprecated.
|
||||
|
||||
In the current version {project_name} will log a warning if the database is not configured to use UTF-8 encoding. Future versions might refuse to work with database that is not configured to use UTF-8 encoding.
|
||||
|
||||
// ------------------------ Removed features ------------------------ //
|
||||
== Removed features
|
||||
|
||||
|
||||
@@ -212,46 +212,60 @@ The database schema provides support for Unicode strings only for the following
|
||||
|
||||
* Descriptions of objects
|
||||
|
||||
Otherwise, characters are limited to those contained in database encoding, which is often 8-bit. However, for some database systems, you can enable UTF-8 encoding of Unicode characters and use the full Unicode character set in all text fields. For a given database, this choice might result in a shorter maximum string length than the maximum string length supported by 8-bit encodings.
|
||||
Otherwise, characters are limited to those contained in database encoding, which is often 8-bit.
|
||||
However, for some database systems, you can enable UTF-8 encoding of Unicode characters and use the full Unicode character set in all text fields.
|
||||
For a given database, this choice might result in a shorter maximum string length than the maximum string length supported by 8-bit encodings.
|
||||
|
||||
=== Configuring Unicode support for an Oracle database
|
||||
The recommended character set to use is summarized in the table below, as most recent versions of these databases already use it by default.
|
||||
|
||||
Unicode characters are supported in an Oracle database if the database was created with Unicode support in the VARCHAR and CHAR fields. For example, you configured AL32UTF8 as the database character set. In this case, the JDBC driver requires no special settings.
|
||||
|
||||
If the database was not created with Unicode support, you need to configure the JDBC driver to support Unicode characters in the special fields. You configure two properties. Note that you can configure these properties as system properties or as connection properties.
|
||||
|
||||
. Set `oracle.jdbc.defaultNChar` to `true`.
|
||||
|
||||
. Optionally, set `oracle.jdbc.convertNcharLiterals` to `true`.
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
For details on these properties and any performance implications, see the Oracle JDBC driver configuration documentation.
|
||||
====
|
||||
|
||||
=== Unicode support for a Microsoft SQL Server database
|
||||
|
||||
Unicode characters are supported only for the special fields for a Microsoft SQL Server database. The database requires no special settings.
|
||||
|
||||
The `sendStringParametersAsUnicode` property of JDBC driver should be set to `false` to significantly improve performance. Without this parameter,
|
||||
the Microsoft SQL Server might be unable to use indexes.
|
||||
|
||||
=== Configuring Unicode support for a MySQL/MariaDB database
|
||||
|
||||
Unicode characters are supported in a MySQL/MariaDB database if the database was created with Unicode support in the VARCHAR and CHAR fields when using the following SQL statement.
|
||||
[%autowidth]
|
||||
|===
|
||||
|Vendor |Recommended Character Set |Example Database Creation
|
||||
|
||||
|Oracle
|
||||
m|AL32UTF8
|
||||
a|
|
||||
[source,sql]
|
||||
----
|
||||
create database keycloak character set utf8mb3;
|
||||
CREATE DATABASE keycloak
|
||||
CHARACTER SET AL32UTF8
|
||||
NATIONAL CHARACTER SET AL16UTF16;
|
||||
----
|
||||
|
||||
Note that the utf8mb4 character set is not supported due to different storage requirements for the utf8 character set. See MySQL documentation for details. In that situation, the length restriction on non-special fields does not apply because columns are created to accommodate the number of characters, not bytes. If the database default character set does not allow Unicode storage, only the special fields allow storing Unicode values.
|
||||
|Microsoft SQL Server
|
||||
|UTF-8 collation +
|
||||
(ending with `_UTF8`)
|
||||
a|
|
||||
[source,sql]
|
||||
----
|
||||
CREATE DATABASE keycloak
|
||||
COLLATE Latin1_General_100_CI_AS_SC_UTF8;
|
||||
----
|
||||
|
||||
. Start MySQL/MariaDB Server.
|
||||
. Under JDBC driver settings, locate the *JDBC connection settings*.
|
||||
. Add this connection property: `characterEncoding=UTF-8`
|
||||
|MySQL/MariaDB
|
||||
m|utf8mb4
|
||||
a|
|
||||
[source,sql]
|
||||
----
|
||||
CREATE DATABASE keycloak
|
||||
CHARACTER SET utf8mb4
|
||||
COLLATE utf8mb4_unicode_ci;
|
||||
----
|
||||
|
||||
==== Database default charset and collation considerations
|
||||
|PostgreSQL
|
||||
m|UTF8
|
||||
a|
|
||||
[source,sql]
|
||||
----
|
||||
CREATE DATABASE keycloak
|
||||
ENCODING 'UTF8'
|
||||
LC_COLLATE 'en_US.utf8'
|
||||
LC_CTYPE 'en_US.utf8';
|
||||
----
|
||||
|
||||
|===
|
||||
|
||||
=== Database default charset and collation considerations for MySQL and MariaDB
|
||||
|
||||
Different MySQL and MariaDB versions may have different default settings for character sets and collations. {project_name} uses the database default charset and collation when creating new tables and columns. This design allows you to manage your own data lifecycle by defining the charset and collation that best fits your requirements.
|
||||
|
||||
@@ -268,25 +282,6 @@ To avoid potential mismatches:
|
||||
By relying on the database default, {project_name} respects your intent and allows you to maintain control over your database configuration throughout its lifecycle.
|
||||
====
|
||||
|
||||
=== Configuring Unicode support for a PostgreSQL database
|
||||
|
||||
Unicode is supported for a PostgreSQL database when the database character set is UTF8. Unicode characters can be used in any field with no reduction of field length for non-special fields. The JDBC driver requires no special settings. The character set is determined when the PostgreSQL database is created.
|
||||
|
||||
. Check the default character set for a PostgreSQL cluster by entering the following SQL command.
|
||||
+
|
||||
[source]
|
||||
----
|
||||
show server_encoding;
|
||||
----
|
||||
|
||||
. If the default character set is not UTF 8, create the database with the UTF8 as the default character set using a command such as:
|
||||
|
||||
+
|
||||
[source]
|
||||
----
|
||||
create database keycloak with encoding 'UTF8';
|
||||
----
|
||||
|
||||
== Preparing for PostgreSQL
|
||||
|
||||
=== Writer and reader instances
|
||||
|
||||
@@ -92,6 +92,7 @@ record JpaInvitationManager(KeycloakSession session, EntityManager em) implement
|
||||
}
|
||||
|
||||
query.where(predicates.toArray(new Predicate[0]));
|
||||
query.orderBy(builder.asc(root.get("id")));
|
||||
|
||||
TypedQuery<OrganizationInvitationEntity> typedQuery = em.createQuery(query);
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ import java.sql.Statement;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Predicate;
|
||||
|
||||
import jakarta.enterprise.inject.Instance;
|
||||
import jakarta.persistence.EntityManager;
|
||||
@@ -103,6 +104,7 @@ public class QuarkusJpaConnectionProviderFactory extends AbstractJpaConnectionPr
|
||||
|
||||
checkMySQLWaitTimeout();
|
||||
checkMSSQLIsolationLevel();
|
||||
checkUtf8Encoding();
|
||||
|
||||
String id = null;
|
||||
String version = null;
|
||||
@@ -386,4 +388,56 @@ public class QuarkusJpaConnectionProviderFactory extends AbstractJpaConnectionPr
|
||||
private static void logErrorSettingMigrationTransactionTimeout(Exception e) {
|
||||
logger.debug("Unable to set the transaction timeout for migration task. Using the default timeout.", e);
|
||||
}
|
||||
|
||||
private void checkUtf8Encoding() {
|
||||
String db = Configuration.getConfigValue(DatabaseOptions.DB).getValue();
|
||||
Database.Vendor vendor = Database.getVendor(db).orElseThrow();
|
||||
switch (vendor) {
|
||||
case TIDB, MARIADB, MYSQL -> checkMySQLUtf8(vendor);
|
||||
case POSTGRES -> checkPostgresEncoding();
|
||||
case MSSQL -> checkMSSQLEncoding();
|
||||
case ORACLE -> checkOracleEncoding();
|
||||
//H2 do we care?
|
||||
}
|
||||
}
|
||||
|
||||
private void checkOracleEncoding() {
|
||||
checkEncoding(Database.Vendor.ORACLE, "AL32UTF8"::equals, "'AL32UTF8' encoding", "SELECT value FROM nls_database_parameters WHERE parameter = 'NLS_CHARACTERSET'");
|
||||
}
|
||||
|
||||
private void checkMSSQLEncoding() {
|
||||
checkEncoding(Database.Vendor.MSSQL, s -> s.endsWith("_UTF8"), "any UTF-8 collation (ending with `_UTF8`)", "SELECT DATABASEPROPERTYEX(DB_NAME(), 'Collation') AS DatabaseCollation");
|
||||
}
|
||||
|
||||
private void checkPostgresEncoding() {
|
||||
checkEncoding(Database.Vendor.POSTGRES, "UTF8"::equals, "'UFT8' encoding", "SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = current_database()");
|
||||
}
|
||||
|
||||
private void checkMySQLUtf8(Database.Vendor vendor) {
|
||||
checkEncoding(vendor, "utf8mb4"::equals, "'utf8mb4' encoding", "SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = DATABASE()");
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the database is running against a database with a valid UTF-8 character encoding.
|
||||
* <p>
|
||||
* Non-UTF-8-character encodings have been deprecated in 26.6.
|
||||
*/
|
||||
private void checkEncoding(Database.Vendor vendor, Predicate<String> isValid, String recommendation, String query) {
|
||||
try (var connection = getConnection();
|
||||
var statement = connection.createStatement();
|
||||
var rs = statement.executeQuery(query)) {
|
||||
rs.next();
|
||||
var encoding = rs.getString(1);
|
||||
if (isValid.test(encoding)) {
|
||||
return;
|
||||
}
|
||||
logInvalidEncoding(vendor, encoding, recommendation);
|
||||
} catch (SQLException e) {
|
||||
logger.warnf(e, "Unable to validate %s encoding due to database exception", vendor);
|
||||
}
|
||||
}
|
||||
|
||||
private static void logInvalidEncoding(Database.Vendor vendor, String encoding, String recommendedEncoding) {
|
||||
logger.warnf("Invalid %s charset encoding '%s'. It is recommended to use %s", vendor, encoding, recommendedEncoding);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,6 +48,7 @@ public abstract class BasicDatabaseTest {
|
||||
@Launch({ "start", AbstractAutoBuildCommand.OPTIMIZED_BUILD_OPTION_LONG, "--http-enabled=true", "--hostname-strict=false" })
|
||||
protected void testSuccessful(CLIResult cliResult) {
|
||||
cliResult.assertStarted();
|
||||
cliResult.assertNoMessage("charset encoding");
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
@@ -84,6 +84,7 @@ public class DatabaseContainer {
|
||||
|
||||
private JdbcDatabaseContainer<?> configureJdbcContainer(JdbcDatabaseContainer<?> jdbcDatabaseContainer) {
|
||||
if (jdbcDatabaseContainer instanceof MSSQLServerContainer) {
|
||||
jdbcDatabaseContainer.withEnv("MSSQL_COLLATION", "Latin1_General_100_CI_AS_SC_UTF8");
|
||||
return jdbcDatabaseContainer;
|
||||
}
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ import org.keycloak.testframework.util.ContainerImages;
|
||||
|
||||
import org.jboss.logging.Logger;
|
||||
import org.testcontainers.containers.JdbcDatabaseContainer;
|
||||
import org.testcontainers.containers.MariaDBContainer;
|
||||
import org.testcontainers.mariadb.MariaDBContainer;
|
||||
import org.testcontainers.utility.DockerImageName;
|
||||
|
||||
class MariaDBTestDatabase extends AbstractContainerTestDatabase {
|
||||
@@ -15,7 +15,7 @@ class MariaDBTestDatabase extends AbstractContainerTestDatabase {
|
||||
|
||||
@Override
|
||||
public JdbcDatabaseContainer<?> createContainer() {
|
||||
return new MariaDBContainer<>(DockerImageName.parse(ContainerImages.getContainerImageName(NAME)).asCompatibleSubstituteFor(NAME)).withCommand("--character-set-server=utf8 --collation-server=utf8_unicode_ci");
|
||||
return new MariaDBContainer(DockerImageName.parse(ContainerImages.getContainerImageName(NAME)).asCompatibleSubstituteFor(NAME));
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -63,7 +63,7 @@ class MSSQLServerTestDatabase extends AbstractContainerTestDatabase {
|
||||
|
||||
@Override
|
||||
public List<String> getPostStartCommand() {
|
||||
return List.of("/opt/mssql-tools18/bin/sqlcmd", "-U", "sa", "-P", getPassword(), "-No", "-Q", "CREATE DATABASE " + getDatabase() + "; " +
|
||||
return List.of("/opt/mssql-tools18/bin/sqlcmd", "-U", "sa", "-P", getPassword(), "-No", "-Q", "CREATE DATABASE " + getDatabase() + " COLLATE Latin1_General_100_CI_AS_SC_UTF8; " +
|
||||
// READ_COMMITTED_SNAPSHOT is recommended for MSSQL to avoid deadlocks
|
||||
"ALTER DATABASE " + getDatabase() + " SET READ_COMMITTED_SNAPSHOT ON;");
|
||||
}
|
||||
|
||||
@@ -519,7 +519,7 @@
|
||||
<docker.database.port>3306</docker.database.port>
|
||||
<docker.database.skip>false</docker.database.skip>
|
||||
<!-- Need to explicitly enable utf8 as it is not the default settings in the mariadb docker image -->
|
||||
<docker.database.cmd>--character-set-server=utf8 --collation-server=utf8_unicode_ci</docker.database.cmd>
|
||||
<docker.database.cmd>--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci</docker.database.cmd>
|
||||
<docker.database.wait-for-log-regex>(?si)Ready for start up.*ready [^\n]{0,30}connections</docker.database.wait-for-log-regex>
|
||||
</properties>
|
||||
</profile>
|
||||
@@ -529,7 +529,7 @@
|
||||
<docker.database.image>${mssql.container}</docker.database.image>
|
||||
<docker.database.port>1433</docker.database.port>
|
||||
<docker.database.skip>false</docker.database.skip>
|
||||
<docker.database.postStart>/opt/mssql-tools18/bin/sqlcmd -e -U sa -P ${keycloak.connectionsJpa.password} -No -d master -Q CREATE\ DATABASE\ ${keycloak.connectionsJpa.database};\ ALTER\ DATABASE\ ${keycloak.connectionsJpa.database}\ SET\ READ_COMMITTED_SNAPSHOT\ ON</docker.database.postStart>
|
||||
<docker.database.postStart>/opt/mssql-tools18/bin/sqlcmd -e -U sa -P ${keycloak.connectionsJpa.password} -No -d master -Q CREATE\ DATABASE\ ${keycloak.connectionsJpa.database}\ COLLATE\ Latin1_General_100_CI_AS_SC_UTF8;\ ALTER\ DATABASE\ ${keycloak.connectionsJpa.database}\ SET\ READ_COMMITTED_SNAPSHOT\ ON</docker.database.postStart>
|
||||
<docker.database.cmd>/bin/sh -c /opt/mssql/bin/sqlservr</docker.database.cmd>
|
||||
<docker.database.wait-for-log-regex>Recovery is complete. This is an informational message only. No user action is required.</docker.database.wait-for-log-regex>
|
||||
<keycloak.storage.connections.vendor>mssql</keycloak.storage.connections.vendor>
|
||||
|
||||
Reference in New Issue
Block a user