mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
1334e4dd52
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15310 Why the change is required: The committed OPTIONS number in MANIFEST gives persisted-options readers a precise source of truth. If that file is missing or corrupt, or an operator must deliberately replace it with a known-good snapshot, persisted-options loading can fail before a normal writable open can publish a replacement. RocksDB therefore needs a supported offline recovery path that obeys the binding and global file-number rules. Previous behavior: ldb update_manifest could rewrite file-temperature metadata but could not repair an OPTIONS binding. Operators could not safely bind a selected snapshot while accounting for the current MANIFEST boundary, every file-number allocation root, recoverable WAL output, and crash durability. Blindly selecting or reusing an existing OPTIONS number could bind the wrong snapshot or collide with another RocksDB file. New behavior: ldb update_manifest --rebind_options_file --options_file=OPTIONS-<number> performs an offline repair from an explicitly selected trusted snapshot whose tracking marker is enabled. Optional --file_number_paths arguments identify external allocation roots that are not serialized in OPTIONS. The command locks the DB and validates that the selected OPTIONS snapshot contains every live MANIFEST column family. Extra OPTIONS CFs are accepted as a safe superset and are filtered by binding-aware persisted-options APIs; a snapshot missing any live CF is rejected. The command raw-copies the selected bytes to a newly allocated canonical OPTIONS number, syncs that file and its directory entry, and commits the new number with one MANIFEST record. It rewrites the current MANIFEST only through the last completely decoded record, appends the commit and next-file boundary to a temporary copy, syncs it, atomically renames it over the original, and syncs the MANIFEST directory. The crash contract has one publication point. Before MANIFEST replacement, the previous commit remains authoritative and the copied tracked OPTIONS file is only an ignorable orphan. After replacement, the new snapshot is authoritative. Implementation: - Parse the selected snapshot for validation but copy it byte-for-byte so unknown or newer option fields are preserved. - Replay MANIFEST to recover live CF membership, the committed number, next file number, and last valid logical-record boundary; discard a torn physical tail during the bounded-memory rewrite. - Scan the DB root, WAL/archive directories, serialized roots, and operator-supplied roots for allocated numbers. - Parse recoverable WALs and reserve conservative headroom for recovery outputs, missing-column-family creation, persistent statistics, asynchronous WAL creation, MANIFEST rollover, and subsequent writable-open allocations. - Reject file-number overflow, missing allocation roots, corrupt or unreadable WALs, invalid input names, missing-live-CF snapshots, and simultaneous temperature rewriting. - Preserve the existing update_temperatures behavior and result text. Safety boundaries and unsupported cases: - This is an explicit offline mutation. It never auto-selects a snapshot and refuses to run while another process holds the DB lock. - The selected input must be a canonical OPTIONS-<number> basename in the DB directory, must parse successfully, must enable MANIFEST tracking, and must contain all live MANIFEST CFs. Extra CFs are allowed but missing live CFs are not. - The command does not repair snapshot contents or alter topology. - OPTIONS does not serialize all DB/CF data roots. Operators must provide every external root through --file_number_paths; the command fails closed when a supplied or inferred root cannot be scanned. - Conservative allocation headroom can reject a repair near kFileNumberMask even when a particular future open would allocate fewer files. - Normal DB open and core publication do not depend on this maintenance command. Validation coverage and known limitations: Coverage includes parsing and incompatible modes, the DB lock, tracking and membership validation, safe-superset acceptance, missing-live-CF rejection, byte-preserving copy, torn MANIFEST tails, one-record commit publication, file-number exhaustion and writable-open headroom, external and missing roots, WAL recovery bounds, asynchronous WAL and persistent-stat allocations, MANIFEST rollover, writable reopen, and later CF creation. The end-to-end test confirms safe-superset rebinding and rejection of a higher tracked snapshot that omits a live CF, verifies the final committed MANIFEST number, and runs the real ldb scan --try_load_options subprocess. Cross-platform CI is not part of local validation. Reviewed By: pdillinger Differential Revision: D123104846 fbshipit-source-id: 8cf7c54c1c7a360411034469614eed8aaffdcf8e
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.