mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
5fe225357a
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15059 **What**: 1. Extends the parallel `CheckpointEngine` so it can also checkpoint a subset of column families. 2. Adds `db_stress` and `db_crashtest.py` coverage for subset checkpoints, over both the serial and the parallel API. **How**: - **Engine API.** `CreateCheckpointOptions` gains a `std::vector<ColumnFamilyHandle*> column_families` field (empty, the default, still means all column families). This goes on the existing per-operation options truct rather than adding a second pure virtual to `CheckpointEngineBase`: it avoids overload-resolution hazards with the defaulted `sequence_number_ptr`, and it is the pattern the engine API already uses. `CreateCheckpointOptions` has no users outside the engine API, so adding a field breaks nothing. - **Shared validation.** The handle checks that were inline in `CheckpointImpl::CreateCheckpoint` -- reject null handles, handles belonging to another DB, and handles for already-dropped families; coalesce ids; always add the default CF -- move to `static CheckpointImpl::ResolveIncludeColumnFamilyIds()`, now called by both APIs. Rejecting a foreign handle matters because `handle->GetID()` alone would silently coerce a foreign id onto whichever family in this DB happens to share it. Validation runs before any staging directory is created, so a rejected request leaves nothing behind. - **Almost no new plumbing.** `CheckpointImpl::CreateCheckpointImpl()` already accepted `include_cf_ids` and is shared by the serial and parallel movers, and the MANIFEST `kColumnFamilyDrop` rewrite runs after `mover->Finish()`, so it was already mover-agnostic. The engine change is just resolving the ids and forwarding them. - **db_stress.** New `subset_cf_checkpoint_one_in` knob (mirrors `parallel_checkpoint_one_in`): each checkpoint operation covers a random proper subset with probability 1/N. It is a no-op when the DB has only the default column family, and it is independent of `parallel_checkpoint_one_in`, so subset checkpoints get exercised through both the serial API and the engine. Subset selection includes each non-default family with probability 1/2 and then force-excludes one, so every subset checkpoint exercises the MANIFEST rewrite instead of degenerating into a whole-DB checkpoint; the default handle is passed in only half the time, covering the "default is always included whether or not you pass it" contract on both sides. - **db_stress validation.** `DB::ListColumnFamilies()` on the checkpoint must return exactly the requested set -- the direct check on the MANIFEST post-processing, catching both an excluded family that was not recorded as dropped and a requested family that went missing. The checkpoint is then opened read-only with exactly those descriptors, proving the reduced MANIFEST and reduced file set are self-consistent under the default `paranoid_checks`. The pre-existing per-key comparison against `SharedState::Exists()` still runs, restricted to the families actually present. - **Incompatibility.** `clear_column_family_one_in > 0` is rejected together with the new knob, the same way D113103899 did for `abort_and_resume_cf_compactions_one_in`: a subset checkpoint hands live handles from `column_families_` to the API and validates against `column_family_names_`, and neither is stable while families are dropped and recreated underneath. `db_crashtest.py` already pins `clear_column_family_one_in` to 0. - **Test cleanup.** The subset tests shared a lot of setup and verification, so `PopulateFourCfsForSubsetCheckpoint()`, `VerifyFourCfsIntact()` and `VerifyDefaultAndTwoSubsetCheckpoint()` were extracted into the test file's anonymous namespace and are reused by the pre-existing `CheckpointSubsetOfCF` as well as the four new engine tests. This extends `StressTest::TestCheckpoint` with a mode rather than adding a separate operation, which is how `parallel_checkpoint_one_in` was integrated. Reviewed By: rban1 Differential Revision: D114832267 fbshipit-source-id: f3f843819614477497cc1466910547922cafc51d
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.