mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
d6a7847d37
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15436 ## Why Long-held global mutexes can stall foreground and background work. Existing mutex statistics measure acquisition waits, not continuous ownership. This adds opt-in logging to identify the mutex, owning thread, operation and release site. ## Enable for an investigation Provide an INFO logger and shared `Statistics` object when opening the DB, then: - Set the shared statistics level to `StatsLevel::kAll`. - Set `mutex_hold_log_threshold_us` through `DB::SetDBOptions`; `100000` means 100 ms. - Restore threshold 0 and the previous statistics level afterward. The threshold defaults to **0 (disabled)**. `kAll` alone does not enable these hold probes; threshold 0 leaves existing statistics timers unchanged. ## Output Illustrative records, not production observations: ``` SLOW_GLOBAL_MUTEX mutex=DBImpl::mutex_ mutex_addr=0x12340000 owner_thread_id=12345 thread_operation=Compaction hold_us=200000 threshold_us=100000 observed_end_us=1792205400000000 buffer_dropped_records=0 unlock=db_impl_compaction_flush.cc:4660 BackgroundCallCompaction() SLOW_GLOBAL_MUTEX mutex=DBImpl::wal_write_mutex_ mutex_addr=0x12340100 owner_thread_id=12346 thread_operation= hold_us=35000 threshold_us=25000 observed_end_us=1792205401000000 buffer_dropped_records=0 scope_exit=db_impl_write.cc:2250 PreprocessWrite() ``` The first proves a completed 200 ms DB-mutex ownership interval on a thread tagged `Compaction`. The second uses a 25 ms threshold; an empty operation tag means none was available. Neither record reports acquisition wait or proves the cause of a stall. Release attribution uses compiler-embedded strings available in optimized, stripped builds, with a GCC/Clang fallback where standard source locations are unavailable. Direct unlock and condition wait identify their call sites; scope exit and temporary unlock identify guard declarations, not closing braces. No stack unwinding or symbol lookup. ## Scope and safety - Track six selected mutexes: DB, WAL, point-lock wait graph, two prepare-log bookkeeping mutexes and transaction name map. This is not comprehensive synchronization tracing. - Acquisition waits and condition-variable release gaps are excluded from hold duration. Configuration is pre-first-use; repeated setup retains the first valid context. - Sample at most once per mutex per second, with eight deferred records per thread. `buffer_dropped_records` counts capacity overflow only; rate limiting, pause and emission contention can omit samples. - Emit after covered outer lock scopes release. Emission is nonblocking best-effort; shutdown drains/pauses emission and stops diagnostics before the owned logger closes. Closing from that logger's own callback is unsupported. - No data-file format change. The mutable option is persisted in OPTIONS and wired into bench/stress tools. Typed C binding is deferred; string DB-option APIs are available. Disabled bookkeeping still has a cost. Release A/B measurements found about 3 ns added per selected mutex pair at level 1, and about 70 ns for hold timing over `kAll`/0. Get and reused-iterator Seek showed no resolved small probe effect; fresh iterator+Seek at 16 threads showed +8.7% CPU/query and -9.4% QPS. No logs were emitted in those benchmarks; logging cost and broader workload qualification remain open. Reviewed By: xingbowang Differential Revision: D125790955 fbshipit-source-id: 1096163f3bd25ffee585dc6ec8ebfd7f1759851a
68 lines
2.6 KiB
C++
68 lines
2.6 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
#pragma once
|
|
|
|
#include <cassert>
|
|
#include <cstdint>
|
|
#include <cstdlib>
|
|
#include <mutex>
|
|
#include <unordered_map>
|
|
#include <vector>
|
|
|
|
#include "monitoring/instrumented_mutex.h"
|
|
#include "rocksdb/rocksdb_namespace.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
// This class is used to track the log files with outstanding prepare entries.
|
|
class LogsWithPrepTracker {
|
|
public:
|
|
// Called when a transaction prepared in `log` has been committed or aborted.
|
|
void MarkLogAsHavingPrepSectionFlushed(uint64_t log);
|
|
// Called when a transaction is prepared in `log`.
|
|
void MarkLogAsContainingPrepSection(uint64_t log);
|
|
// Return the earliest log file with outstanding prepare entries.
|
|
uint64_t FindMinLogContainingOutstandingPrep();
|
|
void ConfigureSlowHoldLogging(
|
|
std::shared_ptr<SlowMutexLoggingContext> context);
|
|
size_t TEST_PreparedSectionCompletedSize() {
|
|
return prepared_section_completed_.size();
|
|
}
|
|
size_t TEST_LogsWithPrepSize() { return logs_with_prep_.size(); }
|
|
|
|
private:
|
|
// REQUIRES: logs_with_prep_mutex_ held
|
|
//
|
|
// sorted list of log numbers still containing prepared data.
|
|
// this is used by FindObsoleteFiles to determine which
|
|
// flushed logs we must keep around because they still
|
|
// contain prepared data which has not been committed or rolled back
|
|
struct LogCnt {
|
|
uint64_t log; // the log number
|
|
uint64_t cnt; // number of prepared sections in the log
|
|
};
|
|
std::vector<LogCnt> logs_with_prep_;
|
|
std::mutex logs_with_prep_mutex_;
|
|
std::unique_ptr<SlowMutexTimer> logs_with_prep_timer_;
|
|
|
|
// REQUIRES: prepared_section_completed_mutex_ held
|
|
//
|
|
// to be used in conjunction with logs_with_prep_.
|
|
// once a transaction with data in log L is committed or rolled back
|
|
// rather than updating logs_with_prep_ directly we keep track of that
|
|
// in prepared_section_completed_ which maps LOG -> instance_count. This helps
|
|
// avoiding contention between a commit thread and the prepare threads.
|
|
//
|
|
// when trying to determine the minimum log still active we first
|
|
// consult logs_with_prep_. while that root value maps to
|
|
// an equal value in prepared_section_completed_ we erase the log from
|
|
// both logs_with_prep_ and prepared_section_completed_.
|
|
std::unordered_map<uint64_t, uint64_t> prepared_section_completed_;
|
|
std::mutex prepared_section_completed_mutex_;
|
|
std::unique_ptr<SlowMutexTimer> prepared_section_completed_timer_;
|
|
};
|
|
} // namespace ROCKSDB_NAMESPACE
|