1. In case of system-versioned table add row_end into FTS_DOC_ID index
in fts_create_common_tables() and innobase_create_key_defs().
fts_n_uniq() returns 1 or 2 depending on whether the table is
system-versioned.
After this patch recreate of FTS_DOC_ID index is required for
existing system-versioned tables. If you see this message in error
log or server warnings: "InnoDB: Table db/t1 contains 2 indexes
inside InnoDB, which is different from the number of indexes 1
defined in the MariaDB" use this command to fix the table:
ALTER TABLE db.t1 FORCE;
2. Fix duplicate history for secondary unique index like it was done
in MDEV-23644 for clustered index (932ec586aa). In case of
existing history row which conflicts with currently inseted row we
check in row_ins_scan_sec_index_for_duplicate() whether that row
was inserted as part of current transaction. In that case we
indicate with DB_FOREIGN_DUPLICATE_KEY that new history row is not
needed and should be silently skipped.
3. Some parts of MDEV-21138 (7410ff436e) reverted. Skipping of
FTS_DOC_ID index for history rows made problems with purge
system. Now this is fixed differently by p.2.
4. wait_all_purged.inc checks that we didn't affect non-history rows
so they are deleted and purged correctly.
Additional FTS fixes
fts_init_get_doc_id(): exclude history rows from max_doc_id
calculation. fts_init_get_doc_id() callback is used only for crash
recovery.
fts_add_doc_by_id(): set max value for row_end field.
fts_read_stopword(): stopwords table can be system-versioned too. We
now read stopwords only for current data.
row_insert_for_mysql(): exclude history rows from doc_id validation.
row_merge_read_clustered_index(): exclude history_rows from doc_id
processing.
fts_load_user_stopword(): for versioned table retrieve row_end field
and skip history rows. For non-versioned table we retrieve 'value'
field twice (just for uniformity).
FTS tests for System Versioning now include maybe_versioning.inc which
adds 3 combinations:
'vers' for debug build sets sysvers_force and
sysvers_hide. sysvers_force makes every created table
system-versioned, sysvers_hide hides WITH SYSTEM VERSIONING
for SHOW CREATE.
Note: basic.test, stopword.test and versioning.test do not
require debug for 'vers' combination. This is controlled by
$modify_create_table in maybe_versioning.inc and these
tests run WITH SYSTEM VERSIONING explicitly which allows to
test 'vers' combination on non-debug builds.
'vers_trx' like 'vers' sets sysvers_force_trx and sysvers_hide. That
tests FTS with trx_id-based System Versioning.
'orig' works like before: no System Versioning is added, no debug is
required.
Upgrade/downgrade test for System Versioning is done by
innodb_fts.versioning. It has 2 combinations:
'prepare' makes binaries in std_data (requires old server and OLD_BINDIR).
It tests upgrade/downgrade against old server as well.
'upgrade' tests upgrade against binaries in std_data.
Cleanups:
Removed innodb-fts-stopword.test as it duplicates stopword.test
This commit fixes the test system hanging due to
the galera_var_notify_ssl_ipv6 test and also brings
the wsrep_notify[_ssl].sh files in line with each other
between the user template and the mtr suite.
Quotes are also added here to avoid problems if the
user specifies the value of one of the variables at the
beginning of the file containing shell-specific characters,
for example, if the password or username specified in the
PSWD and USER variables will contain the "$" character.
Also fixed an issue with automatic --ssl-verify-server-cert
option substitution when the corresponding value is set
by the user to "1" or "on".
Also fixed some tests here to avoid joining one of the nodes
to another cluster when the nodes are restarted from the mtr
side, which can lead to random failures when testing with
buildbot.
Per fsp0types.h, SDI is on tablespace flags position 14 where MariaDB
stores its pagesize. Flag at position 13, also in MariaDB pagesize
flags, is a MySQL encryption flag.
These are checked only if fsp_flags_is_valid fails, so valid MariaDB
pages sizes don't become errors.
The error message "Cannot reset LSNs in table" was rather specific and
not always true to replaced with more generic error.
ALTER TABLE tbl IMPORT TABLESPACE now reports Unsupported on MySQL
tablespace (rather than index corrupted) along with a server error
message.
MySQL innodb Errors are with with UNSUPPORTED rather than CORRUPTED
to avoid user anxiety.
Reviewer: Marko Mäkelä
This commit adds automation that will reduce the possibility
of user errors when customizing wsrep_notify.sh (in particular
caused by user-specified parameters). Now all leading and trailing
spaces are removed from the user-specified parameters and automatic
port and host address substitution has been added to scripts, as
well as automatic password substitution to the client command line,
only if it is specified in the wsrep_notify.sh and not as empty
strings. Also added support for automatic substitution of the all
SSL-related parameters and improved parsing for ipv6 addresses
(to allow "[...]" notation for ipv6 addresses). Also added a
test to check if the wsrep notify script will works with SSL.
1) Removed symlinks that are not very well supported in tar under Windows.
2) Added comment + changed code formatting in viosslfactories.c
3) Fixed a small bug in the yassl code.
4) Fixed a typo in the script code.
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
1. Galera SST scripts should use ssl_capath (not ssl_ca) for CA
directory. The current implementation tries to automatically
detect the path using the trailing slash in the ssl_ca variable
value, but this approach is not compatible with the server
configuration. Now, by analogy with the server, SST scripts
also use a separate ssl_capath variable. In addition, a similar
tcapath variable has been added for the old-style configuration
(in the "sst" section).
2. Openssl utility detection made more reliable.
3. Removed extra spaces in automatically generated command lines -
to simplify debugging of the SST scripts.
4. In general, the code for detecting the presence or absence of
auxiliary utilities has been improved - it is made more reliable
in some configurations (and for shells other than bash).
Changes on top of Sachin’s patch. Specifically:
1) Refined the parsing break condition to only change the parser’s
behavior for parsing strings in binary mode (behavior of \0 outside
of strings is unchanged).
2) Prefixed binary_zero_insert.test with ‘mysql_’ to more clearly
associate the purpose of the test.
3) As the input of the test contains binary zeros (0x5c00),
different text editors can visualize this sequence differently, and
Github would not display it at all. Therefore, the input itself was
consolidated into the test and created out of hex sequences to make
it easier to understand what is happening.
4) Extended test to validate that the rows which correspond to the
INSERTS with 0x5c00 have the correct binary zero data.
Reviewed By:
===========
Andrei Elkin <andrei.elkin@mariadb.com>
Problem:- Some binary data is inserted into the table using Jconnector. When
binlog dump of the data is applied using mysql cleint it gives syntax error.
Reason:-
After investigating it turns out to be a issue of mysql client not able to properly
handle \\\0 <0 in binary>. In all binary files where mysql client fails to insert
these 2 bytes are commom (0x5c00)
Solution:-
I have changed mysql.cc to include for the possibility that binary string can
have \\\0 in it
As MariaDB tables are expected in a fixed order, the cross upgrade
previously placed roles, default_role and max_statement_time after
now unused mysql-5.7 columns.
Test:
$ cp -a /usr/local/mysql-5.7.31/data/ /tmp/m57data
$ sql/mysqld --no-defaults --skip-networking --datadir=/tmp/m57data --socket=/tmp/${PWD##*/}.sock --verbose --lc-messages-dir=$PWD/sql/share
2020-10-02 11:02:05 140135193212864 [Note] sql/mysqld (mysqld 10.2.34-MariaDB) starting as process 1457667 ...
2020-10-02 11:02:05 140135193212864 [Note] InnoDB: Mutexes and rw_locks use GCC atomic built
$ client/mysql_upgrade --no-defaults -u root -pbob -S /tmp/build-mariadb-server-10.2.sock
MySQL upgrade detected
Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv OK
mysql.db OK
mysql.engine_cost OK
mysql.event OK
mysql.func OK
mysql.gtid_executed OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.innodb_index_stats OK
mysql.innodb_table_stats OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.server_cost OK
mysql.servers OK
mysql.slave_master_info OK
mysql.slave_relay_log_info OK
mysql.slave_worker_info OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
Upgrading from a version before MariaDB-10.1
Phase 2/7: Installing used storage engines
Result:
| user | CREATE TABLE `user` (
`Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '',
`User` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
`Password` char(41) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL DEFAULT '',
`Select_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Insert_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Update_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Delete_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Drop_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Reload_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Shutdown_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Process_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`File_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Grant_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`References_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Index_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Alter_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Show_db_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Super_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_tmp_table_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Lock_tables_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Execute_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Repl_slave_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Repl_client_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Show_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Alter_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_user_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Event_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Trigger_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`Create_tablespace_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`ssl_type` enum('','ANY','X509','SPECIFIED') CHARACTER SET utf8 NOT NULL DEFAULT '',
`ssl_cipher` blob NOT NULL,
`x509_issuer` blob NOT NULL,
`x509_subject` blob NOT NULL,
`max_questions` int(11) unsigned NOT NULL DEFAULT 0,
`max_updates` int(11) unsigned NOT NULL DEFAULT 0,
`max_connections` int(11) unsigned NOT NULL DEFAULT 0,
`max_user_connections` int(11) NOT NULL DEFAULT 0,
`plugin` char(64) CHARACTER SET latin1 NOT NULL DEFAULT '',
`authentication_string` text COLLATE utf8_bin NOT NULL,
`password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`is_role` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
`default_role` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
`max_statement_time` decimal(12,6) NOT NULL DEFAULT 0.000000,
`password_last_changed` timestamp NULL DEFAULT NULL,
`password_lifetime` smallint(5) unsigned DEFAULT NULL,
`account_locked` enum('N','Y') COLLATE utf8_bin NOT NULL DEFAULT 'N',
PRIMARY KEY (`Host`,`User`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges' |
MariaDB [(none)]> CREATE ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> SELECT `User`, `is_role` FROM `mysql`.`user`;
+---------------+---------+
| User | is_role |
+---------------+---------+
| root | N |
| mysql.session | N |
| mysql.sys | N |
| dan | N |
| aRole | Y |
+---------------+---------+
5 rows in set (0.00 sec)
Reviewer: Anel Husakovic
On FreeBSD, perl isn't in /usr/bin, its in /usr/local/bin or
elsewhere in the path.
Like storage/{maria/unittest/,}ma_test_* , we use /usr/bin/env to
find perl and run it.
Analysis
Mysqlbinlog output for encrypted binary log
#Q> insert into tab1 values (3,'row 003')
#190912 17:36:35 server id 10221 end_log_pos 980 CRC32 0x53bcb3d3 Table_map: `test`.`tab1` mapped to number 19
# at 940
#190912 17:36:35 server id 10221 end_log_pos 1026 CRC32 0xf2ae5136 Write_rows: table id 19 flags: STMT_END_F
Here we can see Table_map_log_event ends at 980 but Next event starts at 940.
And the reason for that is we do not send START_ENCRYPTION_EVENT to the slave
Solution:-
Send Start_encryption_log_event as Ignorable_log_event to slave(mysqlbinlog),
So that mysqlbinlog can update its log_pos.
Since Slave can request multiple FORMAT_DESCRIPTION_EVENT while master does not
have so We only update slave master pos when master actually have the
FORMAT_DESCRIPTION_EVENT. Similar logic should be applied for START_ENCRYPTION_EVENT.
Also added the test case when new server reads the data from old server which
does not send START_ENCRYPTION_EVENT to slave.
Master Slave Upgrade Scenario.
When Slave is updated first, Slave will have extra logic of handling
START_ENCRYPTION_EVENT But master willnot be sending START_ENCRYPTION_EVENT.
So there will be no issue.
When Master is updated first, It will send START_ENCRYPTION_EVENT to
slave , But slave will ignore this event in queue_event.
This patch allows the server to open old tables that have
"bad" generated columns (i.e. indexed virtual generated columns,
persistent generated columns) that depend on sql_mode,
for general things like SELECT, INSERT, DROP, etc.
Warning are issued in such cases.
Only these commands are now disallowed and return an error:
- CREATE TABLE introducing a "bad" generated column
- ALTER TABLE introducing a "bad" generated column
- CREATE INDEX introdicing a "bad" generated column
(i.e. adding an index on a virtual generated column
that depends on sql_mode).
Note, these commands are allowed:
- ALTER TABLE removing a "bad" generate column
- ALTER TABLE removing an index from a "bad" virtual generated column
- DROP INDEX removing an index from a "bad" virtual generated column
but only if the table does not have any "bad" columns as a result.
for passing ones.
Changes to be committed:
new file: mysql-test/std_data/galera-cert.pem
new file: mysql-test/std_data/galera-key.pem
new file: mysql-test/std_data/galera-upgrade-ca-cert.pem
new file: mysql-test/std_data/galera-upgrade-server-cert.pem
new file: mysql-test/std_data/galera-upgrade-server-key.pem
modified: mysql-test/suite/galera/disabled.def
modified: mysql-test/suite/galera/r/MW-416.result
modified: mysql-test/suite/galera/r/MW-44.result
modified: mysql-test/suite/galera/r/galera_sst_mysqldump_with_key,debug.rdiff
modified: mysql-test/suite/galera/r/galera_sst_mysqldump_with_key.result
modified: mysql-test/suite/galera/t/MW-416.test
modified: mysql-test/suite/galera/t/galera_kill_applier.test
* fix CRL tests to work
* regenerate certificates to be at least 2048 bit
(fixes buster and rhel8 in buildbot)
* update generate-ssl-cert.sh to generate crl files
* make all SSL tests to use certificates generated
in generate-ssl-cert.sh, remove unused certificates
Backport from 10.4 9c60535f86
use frm_version, not mysql_version when parsing frm
In particular, virtual columns are stored according to
frm_version. And CHECK TABLE will overwrite mysql_version
to the current server version, so it cannot correctly
describe frm format.
Test galera checks that a `GRA_x_x.log` file is created whenever
wsrep applier fails to apply some replication event. The file
contains the corresponding binlog event that failed to apply.
The test creates a new file by concatenating a pre-recorded
file containing the binlog header (see `std-data/binlog-header.log`)
and the `GRA_x_x.log` file. The test then checks that the resulting
file, containing the binlog header and the event that failed to
apply, is correctly read by `mysqlbinlog` program.
The test fails in MariaDB because the GRA_x_x.log file created
by MariaDB already contains the binlog header (see MDEV-7867).
This patch fixes/simplifies test `galera.galera_gra_log` so that
it doesn't concatenate `std-data/binlog-header.log` with the
`GRA_x_x.log` file. File `std-data/binlog-header.log` is deleted
altoghether, because not used by any other test.
- Adding a new virtual method Field::load_data_set_no_data().
- Overriding Field_timestamp::load_data_set_no_data() and moving
the TIMESTAMP specific code there.
- Overriding Field_geom::load_data_set_no_data() and implementing
GEOMETRY specific behavior, to prevent writing empty strings
when the loaded file ends unexpectedly. This fixes the bug.
- Adding a new test gis-loaddaata.test.
- The test in loaddata.test for CHAR was added simply to record behavior.
The CHAR data type did not change its behaviour (only GEOMRYRY did).
- Additionally, moving duplicate code into a new method
Field::load_data_set_value() and reusing it in three places.
