mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
176d7c7ea1
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15271 ## Summary Extend the experimental external-table API with a full mode that preserves RocksDB internal keys, sequence numbers, and entry types. Unlike the existing ingestion-oriented mode, full external tables can represent normal flush and compaction output and participate in live databases with puts, merges, deletions, snapshots, `Get`, `MultiGet`, iteration, multiple LSM levels, and external-file ingestion. The existing `ExternalTable*` interfaces remain aliases for the original sequence-zero, Put-only mode, preserving compatibility for current implementations. ## Implementation details - Introduce `ExternalTableMode` and mode-specific reader and factory interfaces. Full-mode builders and iterators receive complete internal keys, while the original mode continues to receive user keys. - Adapt full-mode point lookups to RocksDB’s `GetContext`, allowing readers to return multiple versions of a user key in internal-key order so RocksDB can resolve snapshots, merges, and deletions correctly. - Add a batched `MultiGet` context that preserves per-request lookup state and maps external-reader results back to RocksDB’s original request range. - Translate iterator seek boundaries and returned keys while ensuring file-wide global sequence numbers are applied exactly once. - Track sequence-number bounds, deletion and merge counts, user-collected properties, file size, and unique IDs when building tables. Unsupported entries fail and abandon the builder. - Forward checksum verification to the external reader while retaining compatibility with implementations that do not override it. - Add a standalone test implementation shared by basic and full modes. It uses a simple length-prefixed file format with a RocksDB properties block and checksum, avoiding dependencies on native table implementations. ## Crash testing Add `use_full_external_table` support to `db_stress` and a dedicated blackbox `db_crashtest.py` profile. The profile exercises randomized reads, writes, merges, ingestion, flushes, compactions, snapshots, checksum verification, and crash recovery. The profile disables unsupported combinations, including range deletions, user timestamps, sequence-number time tracking, backward scans, multiple column families, backup reconstruction, remote compaction, dynamic table-factory replacement, randomized aggregate/all-table property probes, and global-sequence-number rewriting during ingestion. Because the simple reader materializes each open table in memory, the profile limits open files and generated file sizes to bound total database memory usage. Reviewed By: xingbowang Differential Revision: D120637416 fbshipit-source-id: aa2cbd06999e4325a234210d9c45987addd37550
23 lines
660 B
CMake
23 lines
660 B
CMake
add_executable(db_stress${ARTIFACT_SUFFIX}
|
|
batched_ops_stress.cc
|
|
cf_consistency_stress.cc
|
|
db_stress.cc
|
|
db_stress_compaction_service.cc
|
|
db_stress_compression_manager.cc
|
|
db_stress_common.cc
|
|
db_stress_driver.cc
|
|
db_stress_filters.cc
|
|
db_stress_gflags.cc
|
|
db_stress_listener.cc
|
|
db_stress_shared_state.cc
|
|
db_stress_test_base.cc
|
|
db_stress_wide_merge_operator.cc
|
|
db_stress_tool.cc
|
|
expected_state.cc
|
|
expected_value.cc
|
|
multi_ops_txns_stress.cc
|
|
no_batched_ops_stress.cc
|
|
../test_util/simple_external_table_factory.cc)
|
|
target_link_libraries(db_stress${ARTIFACT_SUFFIX} ${ROCKSDB_LIB} ${THIRDPARTY_LIBS})
|
|
list(APPEND tool_deps db_stress)
|