Restrict START SLAVE UNTIL to slave options disallowing master options
START SLAVE UNTIL was allowing both slave and master options, which
could cause confusion and misconfigurations. This commit fixes that by
restricting it to these valid slave options:
- MASTER_LOG_FILE, MASTER_LOG_POS
- RELAY_LOG_FILE, RELAY_LOG_POS
- MASTER_GTID_POS
The parser change replaces `master_file_def` with `slave_until_file_def`
in `sql_yacc.yy`, making sure master options like MASTER_USE_GTID and
MASTER_DEMOTE_TO_SLAVE aren’t allowed anymore.
Also expand vcol field index coverings to include indexes covering all
the fields in the expression. The reasoning goes as follows: let f(c1,
c2, ..., cn) be a function on applied to columns c1, c2, ..., cn, if
f(...) is covered by an index, so should vc whose expression is
f(...).
For example, if t.vf = t.c1 + t.c2, and t has three indexes (vf), (c1,
c2), (c1).
Before this change, vf's index covering is a singleton {(vf)}. Let's call
that the "conventional" index covering.
After this change vf's index covering is now {(vf), (c1, c2)}, since
(c1, c2) covers both c1 and c2. Let's call (c1, c2) in this case the
"extra" covering.
With the coverings updated, when an index in the "extra" covering is
chosen for keyread, the vcol also needs to be calculated. In this case
we mark vcol in the table read_set, and ensure it is computed.
With these changes, we see various improvements, including from using
full table scan + filesort to full index scan + filesort when ORDER BY
an indexed vcol (here vc = c + 1 is a vcol and both c and vc are
indexes):
explain select c + 1 from t order by vc;
id select_type table type possible_keys key key_len ref rows Extra
-1 SIMPLE t ALL NULL NULL NULL NULL 10000 Using filesort
+1 SIMPLE t index NULL c 5 NULL 10000 Using index; Using filesort
The substitutions are followed updates to all_fields which include a
copy of the ORDER BY/GROUP BY item pointers, as well as corresponding
updates to ref_pointer_array so that the all_fields and
ref_pointer_array remain in sync.
Another, related change is the recomputation of table index covering
on substitutions. It not only reflects the correct table index
covering after the substitutions, but also improve executions where
the vcol index can be chosen, such as this example (here vc = c + 1
and vc is the only index in the table), from full table scan +
filesort to full index scan:
select vc from t order by c + 1;
We do it in SELECT as well as in single table DELETE/UPDATE.
The buffer is allocated using my_multi_malloc, then updated / filled
in handler calls such as ha_index_first, where the underlying storage
engine may assume it is already initialised and using its values,
causing MSAN to complain. To fix it, we initialise the buffer after it
is allocated.
This fixes MSAN uninitialised failure in gcol.innodb_partition that
may be triggered with a MDEV-36132 patch.
DROP USER looks for sessions by the do-be-dropped user and if found:
* fails with ER_CANNOT_USER in Oracle mode
* continues with ER_ACTIVE_CONNECTIONS_FOR_USER_TO_DROP warning otherwise
Every user being dropped is marked with flag that disallow establishing
a new connections on behalf this user.
let's always disconnect a user connection before dropping the said user.
MariaDB is traditionally very tolerant to active connections
of the dropped user, which isn't the case for most other databases.
Let's avoid unintentionally spreading incompatible behavior
and disconnect before drop.
Except in cases when the test specifically tests such a behavior.
Remove one of the major sources of race condiitons in mariadb-test.
Normally, mariadb_close() sends COM_QUIT to the server and immediately
disconnects. In mariadb-test it means the test can switch to another
connection and sends queries to the server before the server even
started parsing the COM_QUIT packet and these queries can see the
connection as fully active, as it didn't reach dispatch_command yet.
This is a major source of instability in tests and many - but not all,
still less than a half - tests employ workarounds. The correct one
is a pair count_sessions.inc/wait_until_count_sessions.inc.
Also very popular was wait_until_disconnected.inc, which was completely
useless, because it verifies that the connection is closed, and after
disconnect it always is, it didn't verify whether the server processed
COM_QUIT. Sadly the placebo was as widely used as the real thing.
Let's fix this by making mariadb-test `disconnect` command _to wait_ for
the server to confirm. This makes almost all workarounds redundant.
In some cases count_sessions.inc/wait_until_count_sessions.inc is still
needed, though, as only `disconnect` command is changed:
* after external tools, like `exec $MYSQL`
* after failed `connect` command
* replication, after `STOP SLAVE`
* Federated/CONNECT/SPIDER/etc after `DROP TABLE`
and also in some XA tests, because an XA transaction is dissociated from
the THD very late, after the server has closed the client connection.
Collateral cleanups: fix comments, remove some redundant statements:
* DROP IF EXISTS if nothing is known to exist
* DROP table/view before DROP DATABASE
* REVOKE privileges before DROP USER
etc
Statements that intend to modify data have to acquire protection
against ongoing backup. Prior to backup locks, protection against
FTWRL was acquired in form of 2 shared metadata locks of GLOBAL
(global read lock) and COMMIT namespaces. These two namespaces
were separate entities, they didn't share data structures and
locking primitives. And thus they were separate contention
points.
With backup locks, introduced by 7a9dfdd, these namespaces were
combined into a single BACKUP namespace. It became a single
contention point, which doubled load on BACKUP namespace data
structures and locking primitives compared to GLOBAL and COMMIT
namespaces. In other words system throughput has halved.
MDL fast lanes solve this problem by allowing multiple contention
points for single MDL_lock. Fast lane is scalable multi-instance
registry for leightweight locks. Internally it is just a list of
granted tickets, close counter and a mutex.
Number of fast lanes (or contention points) is defined by the
metadata_locks_instances system variable. Value of 1 disables fast
lanes and lock requests are served by conventional MDL_lock data
structures.
Since fast lanes allow arbitrary number of contention points, they
outperform pre-backup locks GLOBAL and COMMIT.
Fast lanes are enabled only for BACKUP namespace. Support for other
namespaces is to be implemented separately.
Lock types are divided in 2 categories: lightweight and heavyweight.
Lightweight lock types represent DML: MDL_BACKUP_DML,
MDL_BACKUP_TRANS_DML, MDL_BACKUP_SYS_DML, MDL_BACKUP_DDL,
MDL_BACKUP_ALTER_COPY, MDL_BACKUP_COMMIT. They are fully compatible
with each other. Normally served by corresponding fast lane, which is
determined by thread_id % metadata_locks_instances.
Heavyweight lock types represent ongoing backup: MDL_BACKUP_START,
MDL_BACKUP_FLUSH, MDL_BACKUP_WAIT_FLUSH, MDL_BACKUP_WAIT_DDL,
MDL_BACKUP_WAIT_COMMIT, MDL_BACKUP_FTWRL1, MDL_BACKUP_FTWRL2,
MDL_BACKUP_BLOCK_DDL. These locks are always served by conventional
MDL_lock data structures. Whenever such lock is requested, fast
lanes are closed and all tickets registered in fast lanes are
moved to conventional MDL_lock data structures. Until such locks
are released or aborted, lightweight lock requests are served by
conventional MDL_lock data structures.
Strictly speaking moving tickets from fast lanes to conventional
MDL_lock data structures is not required. But it allows to reduce
complexity and keep intact methods like: MDL_lock::visit_subgraph(),
MDL_lock::notify_conflicting_locks(), MDL_lock::reschedule_waiters(),
MDL_lock::can_grant_lock().
It is not even required to register tickets in fast lanes. They
can be implemented basing on an atomic variable that holds two
counters: granted lightweight locks and granted/waiting heavyweight
locks. Similarly to MySQL solution, which roughly speaking has
"single atomic fast lane". However it appears to be it won't bring
any better performance, while code complexity is going to be much
higher.
Given that MDL_lock::m_strategy is only accessed by MDL_lock methods,
there is no point in having MDL_lock::needs_notification() and
MDL_lock::hog_lock_types_bitmap() getters anymore.
MDL_lock::has_pending_conflicting_lock() moved to MDL_lock class.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
All MDL_lock objects and types are private now. Along with a set
of methods which are not used outside.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::key from outside of MDL_lock class directly,
use MDL_lock::get_key() instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock members from MDL_context::acquire_lock()
MDL_context::try_acquire_lock_impl() and MDL_map::find_or_insert().
This is done by implementing MDL_map::try_acquire_lock() and
MDL_lock::try_acquire_lock(). With this patch MDL_lock members are
not accessed outside of the class.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
MDL_wait::m_wait_status has to be reset to EMPTY state before going
into waiting routines. There're three options to get it done:
1. Before MDL_lock::rw_lock critical section of
MDL_context::acquire_lock(). In this case MDL_context is not exposed
neither via MDL_lock::m_waiting nor has it MDL_context::m_waiting_for
set.
Cons: m_wait.reset_status() brings unwanted overhead when lock can
be served immediately.
2. Current solution. Within MDL_lock::rw_lock critical section of
MDL_context::acquire_lock(). MDL_context::m_waiting_for is not yet set
however MDL_context was already exposed via MDL_lock::m_waiting list.
The latter is not a problem since we're still holding exclusive lock
on MDL_lock::rw_lock.
Cons: increases critical section size for no good reason.
3. Whenever MDL_wait is created and after wait in
MDL_context::acquire_lock() is completed. At this point MDL_context
is not exposed via MDL_lock::m_waiting anymore and
MDL_context::m_waiting_for is reset.
Cons: none, it is just plain beauty.
Now MDL_wait::m_wait_status is manipulated as following:
EMPTY - set whenever MDL_wait object is created and after each wait
GRANTED - can be set by a thread that releases incompatible lock
VICTIM - can be set either by owner thread or by concurrent higher
priority thread during deadlock detection
TIMEOUT - always set by owner thread
KILLED - always set by owner thread
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_rwlock from MDL_context::try_acquire_lock()
and MDL_context::acquire_lock(), code moved to
MDL_context::try_acquire_lock_impl() instead. It is an intermediate
change that reduce uses of MDL_lock::m_rwlock out of MDL_lock class.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Removed useless MDL_ticket::create() and MDL_ticket::destroy()
indirection. It didn't serve any purpose.
Moved mysql_mdl_create() PFS call out of critical section.
Moved ticket->m_time assignment out of MDL_lock::m_rwlock protection.
It increases critical section size for no good reason. Assigning it
before critical section must be good enough for statistics purposes,
since we must not do long waits here anyway.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_waiting from MDL_context::acquire_lock(),
use MDL_lock::abort_wait() instead.
Avoid accessing MDL_lock::m_granted from MDL_context::release_lock(),
use MDL_lock::release() instead.
Avoid accessing MDL_lock::m_strategy and MDL_lock::m_rwlock from
MDL_map::remove(), code moved to MDL_lock::remove_ticket() instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_rwlock from MDL_context::acquire_lock(),
use MDL_lock::notify_conflicting_locks_if_needed() instead.
Also MDL_lock::needs_notification() doesn't require MDL_lock::m_rwlock
protection, so it is moved out of critical section.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_ticket::upgrade_shared_lock(), use MDL_lock::upgrade() instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_ticket::downgrade_lock(), use MDL_lock::downgrade() instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_granted and MDL_lock::m_rwlock from
MDL_context::clone_ticket(), use MDL_lock::add_cloned_ticket() instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_waiting, MDL_lock::m_granted and
MDL_lock::m_rwlock from mdl_iterate_lock(), use MDL_lock::iterate()
instead.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Avoid accessing MDL_lock::m_rwlock from MDL_map::get_lock_owner(),
code moved to MDL_lock::get_lock_owner() instead.
get_lock_owner() doesn't have to check BACKUP namespace since it is
used by user level locks only.
This is part of broader cleanup, which aims to make large part of
MDL_lock members private. It is needed to simplify further work on
MDEV-19749 - MDL scalability regression after backup locks.
Support for optimizer hints NO_SPLIT_MATERIALIZED and
SPLIT_MATERIALIZED. These hints allow fine-grained control
of the "lateral derived" optimization within a query.
Introduces new overload of hint_table_state function which
tells both a hint's value as well as whether it is present.
This is useful to disambiguate cases that the other version
of hint_table_state cannot, such as when a hint is forcing
a behavior in the optimizer that it would not normally do
and the corresponding optimizer switch is enabled.
When replicating transactions from parallel slave replication
processing, Galera must respect the commit order of the parallel
slave replication. In the current implementation this is done by
calling `wait_for_prior_commit()` before the write set is
replicated and certified in before-prepare processing. This
however establishes a critical section which is held over
whole Galera replication step, and the commit rate will be
limited by Galera replication latency.
In order to allow concurrency in Galera replication step, the
critical section must be released at earliest point where Galera
can guarantee sequential consistency for replicated write sets.
This change passes a callback to release the critical section
by calling `wakeup_subsequent_commits()` to Galera library, which will
call the callback once the correct replication order can be established.
This functionality will be available from Galera 26.4.22 onwards.
Note that call to `wakeup_subsequent_commits()` at this stage is
safe from group commit point of view as Galera uses separate
`wait_for_commit` context to control commit ordering.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
as specified by the comments of this pull request:
https://github.com/MariaDB/server/pull/4078
Also got rid of the file_contents.test and file_contents.result files as
they only test the existence and contents of the INFO_BIN and INFO_SRC,
therefore they are not needed anymore.
Before MySQL 4.0.18, user-specified constraint names were ignored.
Starting with MySQL 4.0.18, the specified constraint name was
prepended with the schema name and '/'. Now we are transforming
into a format where the constraint name is prepended with the
dict_table_t::name and the impossible UTF-8 sequence 0xff.
Generated constraint names will be ASCII decimal numbers.
On upgrade, old FOREIGN KEY constraint names will be displayed
without any schema name prefix. They will be updated to the new
format on DDL operations.
dict_foreign_t::sql_id(): Return the SQL constraint name
without any schemaname/tablename\377 or schemaname/ prefix.
row_rename_table_for_mysql(), dict_table_rename_in_cache():
Simplify the logic: Just rename constraints to the new format.
dict_table_get_foreign_id(): Replaces dict_table_get_highest_foreign_id().
innobase_get_foreign_key_info(): Let my_error() refer to erroneous
anonymous constraints as "(null)".
row_delete_constraint(): Try to drop all 3 constraint name variants.
Reviewed by: Thirunarayanan Balathandayuthapani
Tested by: Matthias Leich
- The test was inadvertently skipped on Windows CI, due to the
unjustified addition of include/have_tlsv13.inc in MDEV-33834 (log TLS
version). That include didn't make sense here and just reduced
coverage.
- Once skipped, the test got broken later by MDEV-12182 changes.
Originally it expected only one localhost:PORT line in the audit log,
assuming Unix socket connections. But on Windows, MTR uses TCP by
default, so all entries had :PORT, and the diff failed.
Fix:
- Forced tcp connection for server_audit.test, via .cnf file
Re-recorded result
unix_socket + server_audit is still covered by other tests.
- Dropped the have_tlsv13.inc include to restore coverage—it wasn't
testing TLS versions or ciphers anyway
Added option 'aria-pagecache-segments', default 1.
For values > 1, this split the aria-pagecache-buffer into the given
number of segments, each independent from each other. Having multiple
pagecaches improve performance when multiple connections runs queries
concurrently using different tables.
Each pagecache will use aria-pageache-buffer/segments amount of
memory, however at least 128K.
Each opened table has its index and data file use the segments in a
a round-robin fashion.
Internal changes:
- All programs allocating the maria pagecache themselves should now
call multi_init_pagecache() instead of init_pagecache().
- pagecache statistics is now stored in 'pagecache_stats' instead of
maria_pagecache. One must call multi_update_pagecache_stats() to
update the statistics.
- Added into PAGECACHE_FILE a pointer to files pagecache. This was
done to ensure that index and data file are using the same
pagecache and simplified the checkpoint code.
I kept pagecache in TABLE_SHARE to minimize the changes.
- really_execute_checkpoint() was update to handle a dynamic number of
pagecaches.
- pagecache_collect_changed_blocks_with_lsn() was slight changed to
allow it to be called for each pagecache.
- undefined not used functions maria_assign_pagecache() and
maria_change_pagecache()
- ma_pagecaches.c is totally rewritten. It now contains all
multi_pagecache functions.
Errors found be QA that are fixed:
MDEV-36872 UBSAN errors in ma_checkpoint.c
MDEV-36874 Behavior upon too small aria_pagecache_buffer_size in case of
multiple segments is not very user-friendly
MDEV-36914 ma_checkpoint.c(285,9): conversion from '__int64' to 'uint'
treated as an error
MDEV-36912 sys_vars.sysvars_server_embedded and
sys_vars.sysvars_server_notembedded fail on x86
MTR .result files currently do not contain output to indicate if a
change_user command has been executed in the corresponding .test files.
Record change_user command in the following format in MTR output only if
disable_query_log is set to false: change_user <user>,<password>,<db>;
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Implements and tests the optimizer hints DERIVED_CONDITION_PUSHDOWN
and NO_DERIVED_CONDITION_PUSHDOWN, table-level hints to enable and
disable, respectively, the condition pushdown for derived tables
which is typically controlled by the condition_pushdown_for_derived
optimizer switch.
Implements and tests the optimizer hints MERGE and NO_MERGE, table-level
hints to enable and disable, respectively, the derived_merge optimization
which is typically controlled by the derived_merge optimizer switch.
Sometimes hints need to be fixed before TABLE instances are available, but
after TABLE_LIST instances have been created (as in the cases of MERGE and
NO_MERGE). This commit introduces a new function called
fix_hints_for_derived_table to allow early hint fixing for derived tables,
using only a TABLE_LIST instance (so long as such hints are not index-level).
Multi-delete code invokes ha_rnd_end after applying deferred deletes, following
the pattern as in multi-update.
The CSV engine batches deletes made via calls to delete_row, then applies them
all at once during ha_rnd_end. For each row batched, the CSV engine decrements
an internal counter of the total rows in the table. The multi-delete code was
not calling ha_rnd_end, so this internal counter was not consistent with the
total number of rows in the table, triggering the assertion.
In the CSV engine, explicitly delete the destination file before renaming the
source file over it. This avoids a file rename problem on Windows. This
change would have been necessary before the latest multi-delete changes, had
this test case existed at that point in time, because the end_read_record call
in do_table_deletes swallowed the rename failure on Windows.
Currently, there are multiple error codes reported for the issue
'Can't execute init_slave query'. Those error codes are the underlying
reason the init_slave query cannot be executed, but this makes it
difficult to detect the issue in an automated way.
This patch introduces a new error code, ER_INIT_SLAVE_ERROR, to unify
all the errors related to the init_slave query. The ER_INIT_SLAVE_ERROR
error is raised for any issue related to the init_slave query, and the
underlying error code and message are included in the Last_SQL_Error
field.
Reviewed by:
Jimmy Hu <jimmy.hu@mariadb.com>
Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Only glibc and not musl currently supports the mechanisms of IFUNC.
This fixes 11.8 branch build on Alpine Linux.
Build error was:
mariadb-11.8.2/sql/vector_mhnsw.cc: In static member function 'static const FVector* FVector::create(metric_type, void*, const void*, size_t)':
mariadb-11.8.2/sql/vector_mhnsw.cc:299:19: error: multiversioning needs 'ifunc' which is not supported on this target
299 | static FVector *align_ptr(void *ptr) { return (FVector*)ptr; }
| ^~~~~~~~~
mariadb-11.8.2/sql/vector_mhnsw.cc:113:3: error: use of multiversioned function without a default
Background:
In MDEV-33474, we introduced runtime dependency packaging primarily to
support libcurl and other potential third-party dependencies from vcpkg.
Problem:
The INSTALL(RUNTIME_DEPENDENCY_SET) command was failing at packaging step
unless shared libraries from the same build were explicitly excluded via
PRE_EXCLUDE_REGEXES. While initially only server.dll was excluded this way,
this turned out insufficient for users compiling their own plugins
Solution:
Exclude all linked shared libraries from the same build via
PRE_EXCLUDE_REGEXES. Move dependency detection and install to the end of
CMake processing, after all add_library/add_executable calls, when all
targets are known.
Also made the INSTALL_RUNTIME_DEPENDENCIES variable independent of vcpkg
detection, for simplicity.
InnoDB does the following check for sequence table during check
table command:
- There should be only one index should exist on sequence table
- There should be only one row should exist on sequence table
- The leaf page must be the root page for the sequence table
- Delete marked record should not exist
- DB_TRX_ID and DB_ROLL_PTR of the record should be 0 and 1U << 55
To check the rows, the table needs to be opened. To that end, and like
MDEV-36038, we force COPY algorithm on ALTER TABLE ... SEQUENCE=1.
This also results in checking the sequence state / metadata.
The table structure was already validated before this patch.
(cherry picked from commit 6f8ef26885)
Two new error codes ER_SEQUENCE_TABLE_HAS_TOO_FEW_ROWS and
ER_SEQUENCE_TABLE_HAS_TOO_MANY_ROWS were introduced in MDEV-36032 in
both 10.11 and, as part of MDEV-22491, 12.0. Here we remove them from
10.11, but they should remain in 12.0.
