mysql-test/suite/rpl/r/last_insert_id.result:
Test case for last_insert_id
mysql-test/suite/rpl/t/last_insert_id.cnf:
Test case for last_insert_id
mysql-test/suite/rpl/t/last_insert_id.test:
Test case for last_insert_id
sql/log_event.cc:
Added DBUG_PRINT
Set thd->first_successful_insert_id_in_prev_stmt_for_binlog when setting thd->first_successful_insert_id_in_prev_stmt.
This is required to get last_insert_id() replicated.
This is analog to how read_first_successful_insert_id_in_prev_stmt() works.
sql/rpl_utility.cc:
Added DBUG_PRINT
The bug was that if mysql.slave_gtid_pos was missing, operations on variables
gtid_slave_pos, gtid_binlog_pos, and gtid_current_pos would fail, and continue
to fail even after the table was fixed, until server restart.
Now setting the variables retry loading the table, succeeding if it has been
restored. And querying the variables when the table is not there acts as if
the table was there and was empty.
Also, attempt to fix a race in the rpl.rpl_gtid_basic test case.
Implement @@gtid_binlog_state. This is the internal state of the binlog
(most recent GTID logged for every domain_id and server_id). This allows
to save the state before RESET MASTER and restore it afterwards.
The ignored events are not written to the relay log, but instead a fake
Rotate event is generated to handle update of position.
Extend this for Gtid so we similarly generate a fake Gtid_list event
to update the GTID position.
Also fix an unrelated test issue that got triggered by the added test cases.
When a GTID event is executed, we remember the contained GTID position so that
when we have applied the entire event group we can commit it to
gtid_slave_pos.
However, if the event group fails to apply due to some error and the SQL
thread aborts, the code did not correctly clear the remembered GTID. Thus,
when SQL thread was restarted, the old GTID of the failing event group was
incorrectly updated to gtid_slave_pos when the initial rotate event was
executed, corrupting the GTID position.
The main bug here was the following situation:
Suppose we set up a completely new master2 as an extra multi-master to an
existing slave that already has a different master1 for domain_id=0. When the
slave tries to connect to master2, master2 will not have anything that slave
requests in domain_id=0, but that is fine as master2 is supposedly meant to
serve eg. domain_id=1. (This is MDEV-4485).
But suppose that master2 then actually starts sending events from
domain_id=0. In this case, the fix for MDEV-4485 was incomplete, and the code
would fail to give the error that the position requested by the slave in
domain_id=0 was missing from the binlogs of master2. This could lead to lost
events or completely wrong replication.
The patch for this bug fixes this issue.
In addition, it cleans up the code a bit, getting rid of the fake_gtid_hash in
the code. And the error message when slave and master have diverged due to
alternate future is clarified, as requested in the bug description.
When a new master is provisioned that does not have any old binlogs,
the @@gtid_slave_pos is used to know where in the GTID history the
provisioning happened. A slave is allowed to connect at the point of
this value of @@gtid_slave_pos, even if that GTID is not in the
binlogs on the new master.
The code to handle this case when the binlog on the newly provisioned
master is completely empty was just wrong (couple of typos). Clearly it
had never been tested ... :-/
When a new master is provisioned that does not have any old binlogs,
the @@gtid_slave_pos is used to know where in the GTID history the
provisioning happened. A slave is allowed to connect at the point of
this value of @@gtid_slave_pos, even if that GTID is not in the
binlogs on the new master.
But --gtid-strict-mode did not correctly handle this case. When strict
mode was enabled, an attempt to connect at the position would cause an
error about holes in the binlog, which is not correct.
This patch adds a hash of GTIDs that need to be treated specially by
GTID strict mode to deal correctly with this case.
The test did RESET MASTER and then tried to use --sync_with_master
to wait for GTID-based replication to catch up. This though has
a race, there is a small window where the _old_ pre-RESET MASTER
position on the slave is higher than the new pos-RESET MASTER
position, causing the --sync_with_master to be a no-op.
Fix by using include/wait_condition.inc instead.
When we load the slave state from the mysql.gtid_slave_pos at server start, we
need to load all the rows into the in-memory hash, not just the most recent
one in each replication domain. Otherwise we accumulate cruft in the form of
old rows each time the server restarts.
In record_gtid(), too many rows were deleted from the slave position
hash - we need to always keep on to the most recent committed row,
so we have a valid slave position at all times.
1. DROP DATABASE should use ha_discover_table_names(), not look at .frm files.
2. filename_to_tablename() also encodes temp file names #sql- -> #mysql50##sql
3. no special treatment for #sql- files, no TABLE_LIST::internal_tmp_table
4. discover also table file names, that start from #
Now whenever we reach the GTID point requested from the slave (when using GTID
position to connect), we send a fake Gtid_list event. This event is used by
the slave to know the current old-style position for MASTER_POS_WAIT(), and
later the similar binlog position for MASTER_GTID_WAIT().
Without this fake event, if the slave is already fully up-to-date with the
master, there may be no events sent at the given position for an indeterminate
time.
If the mysql.gtid_slave_pos table is not available, we cannot load nor update
the current GTID position persistently. This can happen eg. after an upgrade,
before mysql_upgrade_db is run, or if the table is InnoDB and the server is
restarted without the InnoDB storage engine enabled.
Before, replication always failed to start if the table was unavailable. With
this patch, we try to continue with old-style replication, after suitable
complaints in the error log. In strict mode, or if slave is configured to use
GTID, slave still refuses to start.
There was some old code that cleared the position in CHANGE MASTER,
it was forgotten to be removed.
In addition, add code that saves/restores the old-style position
when we nuke the old relay logs as part of GTID slave start.
Normally we will not use these, but it could be useful in case
the GTID connect fails and user wants to go back to the old-style
coordinates.
Fix problems related to reconnect. When we need to reconnect (ie. explict
stop/start of just the IO thread by user, or automatic reconnect due to
loosing network connection with the master), it is a bit complex to correctly
resume at the right point without causing duplicate or missing events in the
relay log. The previous code had multiple problems in this regard.
With this patch, the problem is solved as follows. The IO thread keeps track
(in memory) of which GTID was last queued to the relay log. If it needs to
reconnect, it resumes at that GTID position. It also counts number of events
received within the last, possibly partial, event group, and skips the same
number of events after a reconnect, so that events already enqueued before the
reconnect are not duplicated.
(There is no need to keep any persistent state; whenever we restart slave
threads after both of them being stopped (such as after server restart), we
erase the relay logs and start over from the last GTID applied by SQL thread.
But while the SQL thread is running, this patch is needed to get correct relay
log).
There were several cases where the slave GTID position was not loaded
correctly before being used. This caused various failures such as
corrupting the position at slave start and empty values of
@@gtid_slave_pos and @@gtid_current_pos.
Fixed by adding more checks for loaded position, and by always loading
the position at server startup.
The idea in the code was to protect the user that tries to connect a slave
to a master with completely different domains than what was intended. If
none of the domains in the start position are present at all in the master
binlog, we gave an error.
However, this is a stupid idea. Because when a slave connects to a master
to start replication from the very start of binlogs - such as when setting
up new master->slave servers from scratch - there will be just this
situation, the requested slave position is empty for all the domains in the
master's binlog.
So the code that gives this error is wrong, and the solution is simply to
remove it.
When @@GLOBAL.gtid_strict_mode=1, then certain operations result
in error that would otherwise result in out-of-order binlog files
between servers.
GTID sequence numbers are now allocated independently per domain;
this results in less/no holes in GTID sequences, increasing the
likelyhood that diverging binlogs will be caught by the slave when
GTID strict mode is enabled.
The problem was the Gtid_list event which is logged to the binlog in
10.0 and is not understood by the 5.5 server.
This event is supposed to be replaced with a dummy event for 5.5
servers. But the very first event logged in the very first binlog
has an empty list of GTID, which makes the event too short to be
replacable with an empty event.
The fix is to pad the empty Gtid_list event to be big enough to
be replacable by a dummy event.
Change of user interface to be more logical and more in line with expectations
to work similar to old-style replication.
User can now explicitly choose in CHANGE MASTER whether binlog position is
taken into account (master_gtid_pos=current_pos) or not (master_gtid_pos=
slave_pos) when slave connects to master.
@@gtid_pos is replaced by three separate variables @@gtid_slave_pos (can
be set by user, replicated GTIDs only), @@gtid_binlog_pos (read only), and
@@gtid_current_pos (a combination of the two, most recent GTID within each
domain). mysql.rpl_slave_state is renamed to mysql.gtid_slave_pos to match.
This fixes MDEV-4474.
There was missing a check for THD::killed after THD::enter_cond(). This could
cause the binlog dump thread to miss the kill signal during server shutdown
and hang until it was force-closed.
Also fix a race in a test case that occasionally fails in Buildbot.
Implement START SLAVE UNTIL master_gtid_pos = "<GTID position>".
Add test cases, including a test showing how to use this to promote
a new master among a set of slaves.
- Fixed that wait_timeout_func and wait_timeout tests works on solaris
- We have to compile without NO_ALARM on Solaris as Solaris doesn't support timeouts on sockets with setsockopt(.. SO_RCVTIMEO).
- Fixed that compile-solaris-amd64-debug works (before that we got a wrong ELF class: ELFCLASS64 on linkage)
- Fixed some compiler warnings
- Fixed some failing tests
BUILD/compile-solaris-amd64-debug:
Fixed that compile-solaris-amd64-debug works (before that we got a wrong ELF class: ELFCLASS64 on linkage)
configure.cmake:
We have to compile without NO_ALARM on Solaris as Solaris doesn't support timeouts on sockets with setsockopt(.. SO_RCVTIMEO)
mysql-test/suite/parts/t/partition_basic_innodb.test:
Mark test as big test (as otherwise we get timeout on our opensolaris machine in buildbot)
mysql-test/suite/rpl/t/rpl_cant_read_event_incident.test:
Remove warning
MDEV-4489 "Replication of big5, cp932, gbk, sjis strings makes wrong values on slave"
has been fixed.
Problem:
String constants of some Asian charsets (big5,cp932,gbk,sjis)
can have backslash '\' (0x5C) in the second byte of multi-byte characters.
Replicating of such constants using the standard '\'-escaping is dangerous.
Therefore, constants of these charsets are replicated using hex notation:
INSERT INTO t1 (a) VALUES (0x815C);
However, 0xHHHH constants do not work well in some cases,
because they can behave as strings and as numbers, depending on context
(for example, depending on the data type of the column in an INSERT statement).
This SQL script was not replicated correctly with statement-based replication:
SET NAMES gbk;
PREPARE STMT FROM 'INSERT INTO t1 (a) VALUES (?)';
SET @a = '1';
EXECUTE STMT USING @a;
The INSERT statement was replicated as:
INSERT INTO t1 (a) VALUES (0x31);
'1' was correctly converted to the number 1 on master.
But the 0x31 constant was treated as number 49 on slave.
Fix:
1. Binary log now uses X'HHHH' instead of 0xHHHH constants.
2. The X'HHHH' constants now work always as strings, in all contexts.
This is the SQL standard compliant behaviour.
After the fix, the above statement is replicated as:
INSERT INTO t1 (a) VALUES (X'31');
X'31' is treated as string '1' on slave, and is correctly converted to 1.
modified:
@ mysql-test/r/ctype_cp932_binlog_stm.result
@ mysql-test/r/select.result
@ mysql-test/r/select_jcl6.result
@ mysql-test/r/select_pkeycache.result
@ mysql-test/r/user_var-binlog.result
@ mysql-test/r/varbinary.result
@ mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result
@ mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
@ mysql-test/suite/rpl/r/rpl_charset_sjis.result
@ mysql-test/suite/rpl/r/rpl_mdev382.result
@ mysql-test/suite/rpl/t/rpl_charset_sjis.test
@ mysql-test/t/ctype_cp932_binlog_stm.test
@ mysql-test/t/select.test
@ mysql-test/t/varbinary.test
Adding and updating tests
@ sql/item.cc
@ sql/item.h
@ sql/sql_yacc.yy
@ sql/sql_lex.cc
Splitting the implementations of X'HH' and 0xHH constants into two
separate classes. Fixing the parser to distinguish the two syntaxes.
@ sql/log_event.cc
Using X'HH' instead of 0xHH for binary logging for string constants
of the "dangerous" charsets.
@ sql/sql_string.h
Adding a helped method String::append_hex().
Solaris fixes:
- Fixed that wait_timeout_func and wait_timeout tests works on solaris
- We have to compile without NO_ALARM on Solaris as Solaris doesn't support timeouts on sockets with setsockopt(.. SO_RCVTIMEO).
- Fixed that compile-solaris-amd64-debug works (before that we got a wrong ELF class: ELFCLASS64 on linkage)
- Added missing sync_with_master
Other bug fixes:
- Free memory for rpl_global_gtid_binlog_state before exit() to avoid 'accessing uninitalized mutex' error.
BUILD/FINISH.sh:
Fixed issues on Solaris with ksh
BUILD/compile-solaris-amd64-debug:
Added missing -m64 flag
configure.cmake:
We have to compile without NO_ALARM on Solaris as Solaris doesn't support timeouts on sockets with setsockopt(.. SO_RCVTIMEO)
mysql-test/suite/rpl/t/rpl_gtid_mdev4473.test:
- Added missing sync_with_master (fix by knielsen)
sql-common/client.c:
Added () to get rid of compiler warning
sql/item_strfunc.cc:
Fixed compiler warning
sql/log.cc:
Free memory for static variable rpl_global_gtid_binlog_state before exit()
- If we are compiling with safemalloc, we would try to call sf_free() for some members after sf_terminate() was called, which would result of trying to access the uninitalized mutex 'sf_mutex'
sql/multi_range_read.cc:
Fixed compiler warnings of converting double to ulong.
sql/opt_range.cc:
Fixed compiler warnings of converting double to ulong or uint
- Better to have all variables that can be number of rows as 'ha_rows'
sql/rpl_gtid.cc:
Added rpl_binlog_state::free() to be able to free memory for static objects before exit()
sql/rpl_gtid.h:
Added rpl_binlog_state::free() to be able to free memory for static objects before exit()
sql/set_var.cc:
Fixed compiler warning
sql/sql_join_cache.cc:
Fixed compiler warnings of converting double to uint
sql/sql_show.cc:
Added cast to get rid of compiler warning
sql/sql_statistics.cc:
Remove code that didn't do anything.
(store_record() with record[0] is a no-op)
storage/xtradb/os/os0file.c:
Added __attribute__ ((unused))
support-files/compiler_warnings.supp:
Ignore warnings from atomic_add_64_nv
(was not able to fix this with a cast as the macro is a bit different between systems)
vio/viosocket.c:
Added more DBUG_PRINT
Suppose binlog file X has in its Gtid_list_event: 0-1-3,0-2-5, and suppose the
slave requests to start replicating after 0-1-3.
In this case the bug was that master would start sending events from the start
of X. This is wrong, because 0-2-4 and 0-2-5 are contained in X-1, and are
needed by the slave. So these events were lost.
On the other hand, if the slave requested 0-2-5, then it _is_ correct to start
sending from the beginning of binlog file X, because 0-2-5 is the last GTID
logged in earlier binlogs. The difference is that 0-2-5 is the last of the
GTIDs in the Gtid_list_event. The problem was that the code did not check that
the matched GTID was the last one in the list.
Fixed by checking if the gtid requested by slave that matches a gtid in the
Gtid_list_event is the last event for that domain in the list. If not, go back
to a prior binlog to ensure all needed events are sent to slave.
mysql-test/include/show_events.inc:
Backport --let $binlog_file=LAST, used by MDEV-4473 test case.
This makes it clear that the error code has nothing to do with errno.
mysql-test/include/mtr_warnings.sql:
Fixed suppression for new slave error messages
mysql-test/lib/My/Test.pm:
Use 'send' instead of 'print' to avoid errors about "wrong class ... back attempt"
mysql-test/lib/v1/mtr_report.pl:
Fixed suppression for new slave error messages
mysql-test/mysql-test-run.pl:
Fixed suppression for new slave error messages
Removed warning from perl 5.16.2 about arrays
mysql-test/r/flush_read_lock.result:
Fixed suppression for new slave error messages
sql/rpl_reporting.cc:
Instead of writing "Errcode" to the log for Slave errors, use "Internal MariaDB error code"
The slave dump thread running on the master only checked thd->killed whenever
it reached the end of a binlog file, not between events. This could
unnecessarily delay server shutdown.
This was found by code inspection while tracking down some occasional "forcing
close of thread..." errors in Buildbot. Hopefully this will fix the failures,
but the fix is correct in any case.
Also increase the wait during server shutdown, 2 seconds is a bit tight in
case of heavy I/O stall, and it seems better to delay shutdown a bit than
force-kill threads unnecessarily.
Also fix some races in test cases that restart the mysqld server. The .expect
file should be changed with --append_file, --remove_file + --write_file
creates a short window where mysqld can error out due to .expect file missing.
Merge of 10.0-mdev26 feature tree into 10.0-base.
Global transaction ID is prepended to each event group in the binlog.
Slave connect can request to start from GTID position instead of specifying
file name/offset of master binlog. This facilitates easy switch to a new
master.
Slave GTID state is stored in a table mysql.rpl_slave_state, which can be
InnoDB to get crash-safe slave state.
GTID includes a replication domain ID, allowing to keep track of distinct
positions for each of multiple masters.
The patch contributed by Konstantin Osipov applied.
Native comments:
Implement multiple user-level locks per connection.
GET_LOCK() function in MySQL allows a connection to hold at most
one user level lock. Taking a new lock automatically releases the
old lock, if any.
The limit of one lock per session existed since early versions
of MySQL didn't have a deadlock detector for SQL locks.
MDL patches in MySQL 5.5 added a deadlock detector,
so starting from 5.5 it became possible to take multiple locks
in any order -- a deadlock, should it occur, would be detected
and an error returned to the client which closed the wait chain.
This is exactly what is done in this patch: ULLs are moved
to use MDL subsystem.
* print "table doesn't exist in engine" when a table doesn't exist in the engine,
instead of "file not found" (if no file was involved)
* print a complete filename that cannot be found ('t1.MYI', not 't1')
* it's not an error for a DROP if a table doesn't exist in the engine (or some table
files cannot be found) - if the DROP succeeded regardless
Replace CHANGE MASTER TO ... master_gtid_pos='xxx' with a new system
variable @@global.gtid_pos.
This is more logical; @@gtid_pos is global, not per-master, and it is not
affected by RESET SLAVE.
Also rename master_gtid_pos=AUTO to master_use_gtid=1, which again is more
logical.
Move combining slave and gtid binlog state into a separate function.
Make SHOW ALL SLAVES STATUS use the same function, so it shows the
same value used by slave connect.
Add a test case.
More fixes for test failures in Buildbot:
- Do not run crashing test in Valgrind.
- FLUSH TABLES did not work to avoid errors about not closed tables when
crashing server. Suppress the messages instead.
- Rewrite multi-source test case to only start one pair of slave threads at a
time, to work-around the bug MDEV-4352.
Fix yet another race in the rpl_gtid_startpos test case.
Implement include/wait_for_purge.inc to purge binary logs; we need to
retry the purge multiple times until it succeeds in removing all the logs
we want, as binlog dump threads can be slow to stop on loaded machines and
hold back purge of logs that are still referenced by the thread.
Add tests crashing the slave in the middle of replication and checking that
replication picks-up again on restart in a crash-safe way.
Fix silly code that causes crash by inserting uninitialised data into a hash.
