- Change thd->backup_commit_lock from MDL_request* to MDL_ticket*,
and convert related call sites in handler.cc, log.cc, sql_class.cc,
and xa.cc to use MDL_ACQUIRE_LOCK.
- Convert reload_acl_and_cache() in sql_reload.cc, purge_master_logs()
and reset_master() in sql_repl.cc to MDL_ACQUIRE_LOCK, holding the
ticket in a local MDL_ticket* (these functions used a local
MDL_request originally and should not touch thd->backup_commit_lock).
- Convert acquire_lock() in partition_info.cc to MDL_ACQUIRE_LOCK.
- Add MDL_REQUEST_LIST_ADD() helper for enqueuing lock requests into
MDL_request_list, and convert call sites in sql_base.cc and sp.cc.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
When a replica already has Using_Gtid=No and a CHANGE MASTER TO is
issued with log coordinates (e.g. relay_log_pos, master_log_file),
the server emits a spurious warning:
Note 4190 CHANGE MASTER TO is implicitly changing the value of
'Using_Gtid' from 'No' to 'No'
The value isn't actually changing, so the warning is misleading.
In change_master() (sql/sql_repl.cc), when log coordinates are specified
without an explicit master_use_gtid, the code implicitly sets Using_Gtid
to No and emits a warning. The condition only checks whether
master_use_gtid=No was explicitly given but does not check whether
Using_Gtid is already No.
Added a check that the current Using_Gtid value differs from
USE_GTID_NO before emitting the warning. The warning now only fires
when the value actually changes.
Re-recorded rpl.rpl_from_mysql80 which previously expected the
spurious No-to-No warning.
Reviewed-by: Georgi Kodinov <joro@mariadb.org>
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
https://github.com/MariaDB/server/pull/4678
Replaced macros with optimized inline functions, that use memcpy
instead of unaligned access (translates to single instruction
on architectures we support). Used compiler byteswap intrinsics
where appropriate.
Removed byte_order_generic.h as well as Intel specific headers
byte_order_generic_{x86|x86_64}.h
Added some casts to places where intNstore writes into "const char*".
Master_Server_Id was not cleared after CHANGE MASTER or RESET SLAVE,
showing a stale value until the slave reconnected. Reset master_id
and prev_master_id to 0 in both code paths.
The reset value (0) will be present in SHOW SLAVE STATUS until it is
re-evaluated to the id of a new connected master server.
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Many CHANGE MASTER fields typically have the same
configurations between multi-source connections – namely:
master_connect_retry
master_ssl
master_ssl_ca
master_ssl_capath
master_ssl_cert
master_ssl_cipher
master_ssl_key
master_ssl_verify_server_cert
master_ssl_crl
master_ssl_crlpath
master_use_gtid
master_retry_count
master_heartbeat_period
When MDEV-25674 added `master_retry_count` to CHANGE MASTER, it kept
the server option `--master-retry-count` to be its default value.
This commit back-adds corresponding server options
for the defaults of the rest of those fields.
With them, the command line or config files can set up common
configurations across replication sources (and even replicas).
`--autoset-master-use-gtid` and `--autoset-master-heartbeat-period` can
also reset their prior corresponding options back to their unset states.
CHANGE MASTER can override unset
(defaulted) configurations per connection.
This commit also adds `DEFAULT` keyword support for all of those fields,
so overridden configurations can reset
to the default without RESET REPLICA.
Supporting passing the `DEFAULT` keyword also enables setting
`master_connect_retry` and `master_retry_count` to 0,
which was previously disregarded.
While here, the commit also increases `master_retry_count`
to 64-bit on LLP64 (e.g., Windows) to match LP64 (e.g., Linux).
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
The persistence code of CHANGE MASTER values needs to match file
lines with the formatting for the corresponding field’s type.
This is unstructured, repetitive (not DRY), and makes feature expansions
(such as MDEV-28302 `CHANGE MASTER …=DEFAULT`)
error-prone if not difficult.
This commit moves these functions and global constants from
`slave.cc`/`.h`, as well as the Master and Relay Log Info File
entries from `Rpl_mi` and `Rpl_rli`, to dedicated `rpl_*info_file.h`
files and under corresponding structs to facilitate organization.
Namely, this commit wraps those file entries with transparent
structs that inherit from a `Persistent` interface,
or shared helper structs that themselves inherit from `Persistent`.
By moving the file read/write helper functions to or behind
implementations of `Persistent`’s virtual methods,
reading or writing the file only takes a type-agnostic loop
over the (wrapped) CHANGE MASTER values.
* This commit also includes preemptive support
for preserving MDEV-28302’s `=DEFAULT`.
As such, unset fields (namely `master_connect_retry`)
now remember their `DEFAULT` states
rather than whatever the default is at CHANGE MASTER time.
* For consistency’s sake, `master_heartbeat_period` is
now reset at RESET REPLICA instead of CHANGE MASTER.
As this refactor will disconnect it from fixes for
some open bugs in prior versions, this commit also:
* Reimplements the value reader functions to be strict with their input
* Fixes MDEV-38010
number parsing ignores trailing garbage and overflows
* Supercedes MDEV-38020 integer overflow
* Changes master_heartbeat_period from a `float` of seconds
to a `uint32_t` of milliseconds (adding `/1000.0`s as needed)
* Fixes MDEV-35879 `Slave_heartbeat_period` is imprecise
* The maximum of `master_heartbeat_period` has been
increased to 4294967.295, i.e., (2³²-1)÷1000.
* `master_heartbeat_period` now rounds
instead of truncates (rounds down).
* Prepares to make `master_retry_count` 64-bit on
LLP64 (e.g., Windows) to match LP64 (e.g., Linux)
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Other things:
- Extended mysqltest to write GTID's for master and slave if
sync_slave_with_master fails.
Reviewer: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement an improved binlog implementation that is integrated into
the storage engine. The new implementation is enabled with the
--binlog-storage-engine option. Initially the InnoDB storage engine
implements the binlog.
Integrating the binlog in the storage engine improves performance,
since it makes the InnoDB redo log the single source of truth and
avoids the need for expensive two-phase commit between binlog and
engine. It also makes it possible to disable durability (set
--innodb-flush-log-at-trx-commit=0) to further improve performance,
while still preserving the ability to recover the binlog and database
into a consistent state after a crash.
The new binlog implementation also greatly improves the internal
design and implementation of the binlog, and enables future
enhancements for replication.
This is a squash of the original 11.4-based patch series.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
`LOG_EVENT_OFFSET` is a duplicate of `EVENT_TYPE_OFFSET`
and `BIN_LOG_HEADER_SIZE` depending on the context.
While there, centralize `BIN_LOG_HEADER_SIZE` to `sql/rpl_constants.h`
(previously from `sql/unireg.h` and `client/mysqlbinlog.cc`)
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
CHANGE MASTER can set `Master_Retry_Count` to be lower than
`Connects_Tried`, which’ll look strange to those unaware of that CHANGE.
Now, setting `Master_Retry_Count` also resets `Connects_Tried` to 0.
Reviewed-by: Susil Behera <susil.behera@mariadb.com>
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
When the IO thread (re)connect to a primary,
no updates are available besides unique errors that cause the failure.
These new `Master_info` numbers supplement SHOW SLAVE STATUS’s (most-
recent) ‘Connecting’ state with statistics on (re)connect attempts:
* `Connects_Tried`: how many retries have been attempted so far
This was previously a local variable that only counted re-attempts;
it’s now meaningful even after the “Connecting” state concludes.
* `Master_Retry_Count` (from MDEV-25674): out of how many configured
Side-note: Some of the tests updated by this commit dump the entire
SHOW SLAVE STATUS, which might include non-deterministic entries.
Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This new CHANGE MASTER TO field specifies the `--master-retry-count`
(global option: the number of Primary connection attempts)
for each multi-source replica; i.e, per-channel `performance_schema.`
`replication_connection_configuration.CONNECTION_RETRY_COUNT`.
`--master-retry-count` remains the default for new `CHANGE MASTER TO`s.
This new keyword and `master-info` entry
matches those of pre-‘REPLICATION SOURCE’ MySQL.
* rpl.rpl_system_versioning_partitions updated for MDEV-32188
* innodb.row_size_error_log_warnings_3 changed error for MDEV-33658
(checks are done in a different order)
In Log_event::read_log_event(), don't use IO_CACHE::error of the relay log's
IO_CACHE to signal an error back to the caller. When reading the active
relay log, this flag is also being used by the IO thread, and setting it can
randomly cause the IO thread to wrongly detect IO error on writing and
permanently disable the relay log.
This was seen sporadically in test case rpl.rpl_from_mysql80. The read
error set by the SQL thread in the IO_CACHE would be interpreted as a
write error by the IO thread, which would cause it to throw a fatal
error and close the relay log. And this would later cause CHANGE
MASTER to try to purge a closed relay log, resulting in nullptr crash.
SQL thread is not able to parse an event read from the relay log. This
can happen like here when replicating unknown events from a MySQL master,
potentially also for other reasons.
Also fix a mistake in my_b_flush_io_cache() introduced back in 2001
(fa09f2cd7e) where my_b_flush_io_cache() could wrongly return an error set
in IO_CACHE::error, even if the flush operation itself succeeded.
Also fix another sporadic failure in rpl.rpl_from_mysql80 where the outout
of MASTER_POS_WAIT() depended on timing of SQL and IO thread.
Reviewed-by: Monty <monty@mariadb.org>
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
cannot have an assert in Warning_info::push_warning()
because SQL command SIGNAL can set an absolutely arbitrary
message, even an empty one or ending with '\n'
move the assert into push_warning() and my_message_sql().
followup for 9508a44c37
MDEV-29533 Crash when MariaDB is replica of MySQL 8.0
MySQL 8.0 has added the following new events in the MySQL binary log
PARTIAL_UPDATE_ROWS_EVENT
TRANSACTION_PAYLOAD_EVENT
HEARTBEAT_LOG_EVENT_V2
- PARTIAL_UPDATE_ROWS_EVENT is used by MySQL to generate update
statements using JSON_SET, JSON_REPLACE and JSON_REMOVE to make
update of JSON columns more efficient. These events can be
disabled by setting 'binlog-row-value-options=""'
- TRANSACTION_PAYLOAD_EVENT is used by MySQL to signal that a
row event is compressed. It an be disably by setting
'binlog_transaction_compression=0'.
- HEARTBEAT_LOG_EVENT_V2 is written to the binary log many times
per seconds. It can be ignored by the server.
What this patch does:
- If PARTIAL_UPDATE_ROWS_EVENT or TRANSACTION_PAYLOAD_EVENT is found,
the server will stop with an error message of how to disable the
MySQL server to generate such events.
- HEARTBEAT_LOG_EVENT_V2 events are ignored.
- mariadb-binlog will write the name of the new events.
- mariadb-binlog will stop if PARTIAL_UPDATE_ROWS_EVENT or
TRANSACTION_PAYLOAD_EVENT is found, unless --force is given.
- Fixes a crash in mariadb-binlog if a character set unknown to
MariaDB is found. (MDEV-29533)
From Kristian Nielsen:
- Add test case for MySQL 8.0 to MariaDB replication and fixed a
a small typo in post_header_len initialization.
Reviewer: knielsen@mariadb.org
This commit adds 3 new status variables to 'show all slaves status':
- Master_last_event_time ; timestamp of the last event read from the
master by the IO thread.
- Slave_last_event_time ; Master timestamp of the last event committed
on the slave.
- Master_Slave_time_diff: The difference of the above two timestamps.
All the above variables are NULL until the slave has started and the
slave has read one query event from the master that changes data.
- Added information_schema.slave_status, which allows us to remove:
- show_master_info(), show_master_info_get_fields(),
send_show_master_info_data(), show_all_master_info()
- class Sql_cmd_show_slave_status.
- Protocol::store(I_List<i_string_pair>* str_list) as it is not
used anymore.
- Changed old SHOW SLAVE STATUS and SHOW ALL SLAVES STATUS to
use the SELECT code path, as all other SHOW ... STATUS commands.
Other things:
- Xid_log_time is set to time of commit to allow slave that reads the
binary log to calculate Master_last_event_time and
Slave_last_event_time.
This is needed as there is not 'exec_time' for row events.
- Fixed that Load_log_event calculates exec_time identically to
Query_event.
- Updated RESET SLAVE to reset Master/Slave_last_event_time
- Updated SQL thread's update on first transaction read-in to
only update Slave_last_event_time on group events.
- Fixed possible (unlikely) bugs in sql_show.cc ...old_format() functions
if allocation of 'field' would fail.
Reviewed By:
Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Kristian Nielsen <knielsen@knielsen-hq.org>
PURGE BINARY LOGS did not always purge binary logs. This commit fixes
some of the issues and adds notifications if a binary log cannot be
purged.
User visible changes:
- 'PURGE BINARY LOG TO log_name' and 'PURGE BINARY LOGS BEFORE date'
worked differently. 'TO' ignored 'slave_connections_needed_for_purge'
while 'BEFORE' did not. Now both versions ignores the
'slave_connections_needed_for_purge variable'.
- 'PURGE BINARY LOG..' commands now returns 'note' if a binary log cannot
be deleted like
Note 1375 Binary log 'master-bin.000004' is not purged because it is
the current active binlog
- Automatic binary log purges, based on date or size, will write a
note to the error log if a binary log matching the size or date
cannot yet be deleted.
- If 'slave_connections_needed_for_purge' is set from a config or
command line, it is set to 0 if Galera is enabled and 1 otherwise
(old default). This ensures that automatic binary log purge works
with Galera as before the addition of
'slave_connections_needed_for_purge'.
If the variable is changed to 0, a warning will be printed to the error
log.
Code changes:
- Added THD argument to several purge_logs related functions that needed
THD.
- Added 'interactive' options to purge_logs functions. This allowed
me to remove testing of sql_command == SQLCOM_PURGE.
- Changed purge_logs_before_date() to first check if log is applicable
before calling can_purge_logs(). This ensures we do not get a
notification for logs that does not match the remove criteria.
- MYSQL_BIN_LOG::can_purge_log() will write notifications to the user
or error log if a log cannot yet be removed.
- log_in_use() will return reason why a binary log cannot be removed.
Changes to keep code consistent:
- Moved checking of binlog_format for Galera to be after Galera is
initialized (The old check never worked). If Galera is enabled
we now change the binlog_format to ROW, with a warning, instead of
aborting the server. If this change happens a warning will be printed to
the error log.
- Print a warning if Galera or FLASHBACK changes the binlog_format
to ROW. Before it was done silently.
Reviewed by: Sergei Golubchik <serg@mariadb.com>,
Kristian Nielsen <knielsen@knielsen-hq.org>
Similar to #2480.
567b681 introduced safe_strcpy() to minimize the use of C with
potentially unsafe memory overflow with strcpy() whose use is
discouraged.
Replace instances of strcpy() with safe_strcpy() where possible, limited
here to files in the `sql/` directory.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
in the $case=2 - it's wrong to kill after the first binlog EOF,
because that might happen between INSERT(4) and INSERT(5).
So, wait for the slave to acknowledge INSERT(5) before killing
the master, that is, both connection threads must pass
repl_semisync_master.wait_after_sync()
