mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
9475798205
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15111 T284514290 exposed an online-recovery race between the automatic recovery coordinator and a foreground synchronous write. A retryable no-WAL flush error selected lightweight retry-flush recovery. While that recovery attempt waited for the failed background flush, `Delete(sync=true)` appended and synced sequence N to the active WAL, then failed while persisting WAL-tracking metadata to the MANIFEST. Because memtable insertion and sequence publication happen after that MANIFEST step, the record was durable but logically unpublished. OLD: the concurrent error triggered another attempt, but the attempt reused a stale recovery plan. That preserved both ingredients needed for a duplicate: the first `Delete(key, N)` remained in the active WAL, while the published sequence remained `N-1`. ``` Recovery coordinator Foreground sync-write thread ------------------------------------ -------------------------------- [db_mutex held] plan = RetryFlush (copied once before the retry loop) attempt 1: ResumeImpl(plan) WaitForBackgroundWork() bg_cv_.Wait() [atomically releases db_mutex] ---------> assign sequence N append + sync Delete(key, N) [db_mutex held] MANIFEST WAL edit fails no memtable insert no LastSequence publication write returns I/O error recovery_error_ = Hard (requests another attempt) bg_error_ = Hard recover_context_ eventually becomes FullRecovery [db_mutex released] [db_mutex reacquired] <--------- ClearBGError() sees recovery_error_ attempt 1 cannot clear the error attempt 2: recovery_error_ = OK ResumeImpl(plan) ^ +-- still the cached RetryFlush plan RetryFlushesForErrorRecovery() retries the failed immutable flush only does not call SwitchMemtable() does not switch the active WAL ClearBGError() succeeds caller retries Delete(key) LastSequence is still N-1 => assigns sequence N again appends Delete(key, N) to same WAL memtable insert succeeds publishes LastSequence = N Reopen / WAL replay: first (key, N, deletion) -> memtable insert succeeds second (key, N, deletion) -> TryAgain("key+seq exists") normal WAL recovery has seq_per_batch_ == false assert(seq_per_batch_) fails ``` The cached argument to `ResumeImpl()` is therefore the direct bridge to the assertion: lightweight retry-flush recovery can report success without retiring the WAL that contains the unpublished record. Since it also does not publish sequence N, the successful caller retry writes the identical internal key into that same WAL. NEW: the same recovery coordinator consumes the concurrent error through a mutex-ordered handoff. ``` Recovery coordinator Foreground sync-write thread ------------------------------------ -------------------------------- [db_mutex held] attempt 1: plan = recover_context_ // RetryFlush ResumeImpl(plan) WaitForBackgroundWork() bg_cv_.Wait() [atomically releases db_mutex] ---------> append + sync Delete(key, seq=N) [db_mutex held] MANIFEST WAL edit fails CheckAndSet...(Hard, FullRecovery): recovery_error_ = Hard bg_error_ = Hard recover_context_ = FullRecovery [db_mutex released] [db_mutex reacquired] <--------- ClearBGError() sees recovery_error_ current attempt cannot clear the error next retry, with db_mutex held: recovery_error_ = OK plan = recover_context_ // FullRecovery ResumeImpl(plan) SwitchMemtable() switch to a new WAL clear the background error ``` The foreground unlock of `db_mutex_` happens before the recovery coordinator reacquires it and snapshots the next plan. Publishing `bg_error_` and `recover_context_` together under that mutex prevents the next attempt from observing a new severity with an old recovery requirement. `recovery_error_` is the per-attempt latch: a concurrent I/O error cannot be cleared as part of the in-flight attempt; it either stops recovery or forces another attempt to consume the updated context. If another error arrives while `ResumeImpl()` releases the mutex again, the same handoff repeats. No second recovery coordinator is needed. Manual `Resume()` now uses the retained recovery context instead of reconstructing one from the background-work scheduling flag, and the soft-error clear-only shortcut is allowed only when that flag shows there is no pending recovery work. The write ordering and `assert(seq_per_batch_)` detector are unchanged. Reviewed By: anand1976, xingbowang Differential Revision: D116106177 fbshipit-source-id: a78450c0924a326b5302ae6709e367f07bd3d5a6
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.