mirror of
https://github.com/MariaDB/server.git
synced 2026-10-23 18:49:29 +03:00
08377b68b1
for unnecessarily long time Issue: dict_load_table_one() was invoked with the exclusive dict_sys.latch held and never released it. The whole load ran inside that one critical section: reading the SYS_TABLES record, opening the .ibd file, and reading SYS_COLUMNS, SYS_VIRTUAL, SYS_INDEXES, SYS_FIELDS, the clustered index root page, and SYS_FOREIGN, as well as loading every table related by FOREIGN KEY constraints. Almost all of that is I/O. Because dict_sys.latch is the single latch that guards every dictionary lookup, one cold table open blocked every other session from opening any table, including tables that were already cached. A slow enough load could trigger the fatal semaphore wait check. Fix: Split the load into three phases. A short latched phase creates the table object and publishes it as an incomplete "stub"; the I/O runs with no latch held; a final latched phase links the FOREIGN KEY constraints. The stub's progress is held in the top two bits of dict_table_t:: n_ref_count (LOADING_DEF, LOADING_FK, LOAD_FAILED), analogous to how buf_page_t::state() combines a small lifecycle state with the buffer-fix count in one atomic. A thread that finds a loading table waits by pinning it (dict_table_t::try_pin_for_wait(), so it cannot be freed while unlatched) and acquiring dict_table_t::lock_latch in shared mode, the same latch the loader holds in exclusive mode for the duration of the load. dict_sys_t::wait_for_load() reports whether the load it waited for actually succeeded, so a caller that only needed to wait for the table it already found can use it directly afterwards instead of repeating the lookup; a caller that only wanted to confirm a related table had finished linking its constraints can likewise skip straight to treating it as already linked. A failed load can be torn down safely even though other threads may have already taken such a pin to wait on it: try_pin_for_wait() refuses once LOAD_FAILED is set, and the teardown path drains any pins taken earlier before freeing the stub. A table remains hidden (LOADING_FK) until every table related to it by FOREIGN KEY constraints has been loaded as well, and dict_sys_t:: load_table() makes them all visible in one step. The intermediate state is visible to constraint linking via dict_sys_t::find_table_fk(), so that two threads loading tables that reference each other can still link the constraint between them. Tables that InnoDB internal SQL can refer to are loaded without ever releasing the latch. The internal SQL parser is not reentrant and is serialized only by the exclusive dict_sys.latch, and it opens tables in the middle of parsing; releasing the latch there let another thread run the parser concurrently and corrupt its state. Changes: - dict_table_t : hold the progress of loading in the top bits of n_ref_count (std::atomic<uint32_t>): LOADING_DEF, LOADING_FK, LOAD_FAILED, loading(). Add the loader-only helpers start_loading(), advance_to_loading_fk(), load_finish(), mark_load_failed(), and try_pin_for_wait() for waiters, all built on the existing lock_latch. - dict_load_table_one() : publish the table as a LOADING_DEF stub in table_non_LRU and release the latch before loading the tablespace, the columns and the indexes; reacquire it, move the table to table_LRU and advance it to LOADING_FK before loading the foreign key constraints. - dict_sys_t::load_table() : wait for a concurrent load of the same table via wait_for_load(); allocate the foreign key table names on a local heap, because the latch is released while draining them; in the drain loop, wait for a table whose definition is still being loaded by another thread and skip one that is only waiting for its own related tables; make all tables loaded by this invocation visible in one step via load_finish(). - dict_sys_t::wait_for_load() : pins the observed table via try_pin_for_wait() (returning false immediately if the load already failed), releases dict_sys.latch, blocks on the table's own lock_latch, reacquires dict_sys.latch, and only then checks loading() and releases its pin, returning whether the load succeeded. Because the latch is reacquired before the pin is released, a successful wait leaves the table valid to use directly, with no need to repeat the lookup that found it. Takes an excl parameter: whether the caller holds dict_sys.latch in exclusive mode (lock()/unlock()) or shared mode (freeze()/unfreeze()), so it knows which pair to use to release and reacquire it. - dict_load_table_one_discard() : used wherever a stub must be torn down (retry after DB_SUCCESS_LOCKED_REC, column or virtual-column load failure, a corrupted index or missing FK index). Calls mark_load_failed(), which marks the stub LOAD_FAILED and releases lock_latch to wake any already-pinned waiters, then drains the reference count to zero before removing the stub. - dict_sys_t::add() : take lock_latch in exclusive mode on the loader's behalf before a loading stub becomes reachable via find_table_any(). - dict_get_and_save_data_dir_path() : skip re-acquiring lock_latch when the table is loading(), because the loading thread already holds it in exclusive mode and is the only thread that can reach the table at that point. - dict_load_hold_latch() : whether a table may be referenced by InnoDB internal SQL, and therefore must be loaded without releasing the latch: InnoDB system tables, FULLTEXT INDEX auxiliary tables and the persistent statistics tables. The FULLTEXT prefix match is case-sensitive, matching how these auxiliary table names are actually generated. - dict_load_table_on_id() : copy the table name and release the SYS_TABLES page latch before calling load_table(), which may now block. Restore the cursor position only if the scan has to continue. - dict_load_foreign(), dict_load_foreigns() : add the fk_heap parameter and allocate the names appended to fk_tables on it. - dict_sys_t::find_table_any() : an unconditional lookup by table name or table_id_t, returning a table even while it is still being loaded. Only the name (or id) and hash pointers may be read on such a table. - dict_sys_t::find_table() : wait for a concurrent load of the table it finds, via wait_for_load(), both in the by-name and the by-id variant, retrying the lookup if the load failed, since the stub may have been freed, or a fresh load of the same name or id may now be in progress. Takes an excl parameter so that the caller can state which dict_sys.latch mode it holds: dict_acquire_mdl_shared(), dict_table_open_on_id(), dict_table_open_on_name(), row_purge_remove_clust_if_poss_low(), trx_t::evict_table(), UndorecApplier::apply_undo_rec(), trx_purge_table_open(), prepare_inplace_alter_table_dict(), lock_release(), and create_table_info_t::create_foreign_keys() all pass it. - dict_sys_t::find_table_fk() : like find_table(), but LOADING_FK tables are returned, so that constraints can be linked into them while holding the exclusive latch. - dict_table_can_be_evicted() : a table that is being loaded may only be freed by the thread that is loading it. - dict_foreign_add_to_cache() : resolve both sides with find_table_fk(). - dict_table_t::load_failed_refs_cleared() : whether every reference taken before a failed load has been released. dict_load_table_one_discard() uses this to drain a failed stub before freeing it, instead of comparing get_ref_count() to 0. - create_table_info_t::create_foreign_keys() : take a temporary reference on a referenced table as soon as it is resolved, released on every exit path by a scope guard; call dict_sys.prevent_eviction() only once the constraint is actually committed to the dictionary cache, replacing the temporary reference. - create_table_info_t::create_table() : acquire a reference to the created table around the foreign key handling, and use a local heap for the names of the foreign key related tables. - assertion fix : the load path may now run without dict_sys.latch, but only in the thread that is loading the table, and a cached table is no longer necessarily fully loaded. dict_sys.locked() is relaxed to "dict_sys.locked() || table->is_loader()" in dict_load_columns(), dict_load_virtual_col(), dict_load_fields(), dict_load_indexes(), dict_index_add_to_cache(), dict_index_find_cols(), dict_index_build_internal_clust(), dict_index_build_internal_non_clust() and dict_index_build_internal_fts(). Checks of dict_table_t::cached are relaxed and reordered after the atomic loading in dict_table_add_system_columns(), dict_sys_t::add(), dict_table_rename_in_cache() and hash_insert(). - innodb.dict_load_concurrent A load is parked at dict_load_table_one_no_latch while holding no latch; another table can be loaded meanwhile, and a second opener of the same table waits. A second case checks that a table is not made visible while a table related to it by a FOREIGN KEY constraint is still being loaded by another thread.
Code status:
ci.appveyor.com
MariaDB: The innovative open source database
MariaDB was designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.
MariaDB is brought to you by the MariaDB Foundation and the MariaDB Corporation. Please read the CREDITS file for details about the MariaDB Foundation, and who is developing MariaDB.
MariaDB is developed by many of the original developers of MySQL who now work for the MariaDB Corporation, the MariaDB Foundation and by many people in the community.
MySQL, which is the base of MariaDB, is a product and trademark of Oracle Corporation, Inc. For a list of developers and other contributors, see the Credits appendix. You can also run 'SHOW authors' to get a list of active contributors.
A description of the MariaDB project and a manual can be found at:
https://mariadb.com/docs/release-notes
Getting the code, building it and testing it
Refer to the following guide: https://mariadb.org/get-involved/getting-started-for-developers/get-code-build-test/ which outlines how to build the source code correctly and run the MariaDB testing framework, as well as which branch to target for your contributions.
Help
More help is available from the Maria Discuss mailing list https://lists.mariadb.org/postorius/lists/discuss.lists.mariadb.org/ and MariaDB's Zulip instance, https://mariadb.zulipchat.com/
Licensing
MariaDB is specifically available only under version 2 of the GNU General Public License (GPLv2). (I.e. Without the "any later version" clause.) This is inherited from MySQL. Please see the README file in the MySQL distribution for more information.
License information can be found in the COPYING file. Third party license information can be found in the THIRDPARTY file.
Bug Reports
Bug and/or error reports regarding MariaDB should be submitted at: https://jira.mariadb.org
For reporting security vulnerabilities, see our security-policy.
The code for MariaDB, including all revision history, can be found at: https://github.com/MariaDB/server