Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15334
External table formats cannot currently persist range tombstones, so SST ingestion and full-mode DB flushes or compactions fail when they contain `DeleteRange` entries. This change adds an opt-in contract for storing RocksDB’s canonical range-deletion block. RocksDB retains responsibility for sequence-number handling, fragmentation, and visibility through its native range-tombstone iterators.
Existing factories remain unsupported by default, and existing files without range tombstones are unaffected. Missing or invalid advertised tombstone blocks are treated as corruption. RocksDB also supplies logical entry counts and sizes when a format does not persist the properties block, while preserving format-specific properties.
Reviewed By: xingbowang
Differential Revision: D123475827
fbshipit-source-id: de8fec401bdc311282f5d4a48b248875c682f7c9
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15287
`IngestExternalFileOptions::write_global_seqno` has been deprecated since RocksDB 8.0 and will soon be removed from the API. It requires ingestion to mutate SST files, sync them, regenerate checksums, and expose file-offset plumbing solely for that mutation.
Remove the option and its C, Java, and `ldb` APIs. Assigned sequence numbers now live only in the MANIFEST, leaving newly ingested SST bytes and whole-file checksums unchanged.
## Backward Compatibility
New readers continue to support existing SSTs containing a nonzero global sequence number written by legacy ingestion. Because that old write did not update the properties-block checksum, checksum verification retries against a temporary copy with only the global-seqno value restored to its original zero placeholder. The parsed sequence number and file contents remain unchanged.
This fallback only accepts mismatches explained by the legacy rewrite. Other checksum failures continue through normal corruption handling, and `verify_checksums=false` continues to bypass verification.
Reviewed By: xingbowang
Differential Revision: D122405856
fbshipit-source-id: 934e9bc755d58ea8d84e1c085e9769015bd8dfd7
Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15271
## Summary
Extend the experimental external-table API with a full mode that preserves RocksDB internal keys, sequence numbers, and entry types. Unlike the existing ingestion-oriented mode, full external tables can represent normal flush and compaction output and participate in live databases with puts, merges, deletions, snapshots, `Get`, `MultiGet`, iteration, multiple LSM levels, and external-file ingestion.
The existing `ExternalTable*` interfaces remain aliases for the original sequence-zero, Put-only mode, preserving compatibility for current implementations.
## Implementation details
- Introduce `ExternalTableMode` and mode-specific reader and factory interfaces. Full-mode builders and iterators receive complete internal keys, while the original mode continues to receive user keys.
- Adapt full-mode point lookups to RocksDB’s `GetContext`, allowing readers to return multiple versions of a user key in internal-key order so RocksDB can resolve snapshots, merges, and deletions correctly.
- Add a batched `MultiGet` context that preserves per-request lookup state and maps external-reader results back to RocksDB’s original request range.
- Translate iterator seek boundaries and returned keys while ensuring file-wide global sequence numbers are applied exactly once.
- Track sequence-number bounds, deletion and merge counts, user-collected properties, file size, and unique IDs when building tables. Unsupported entries fail and abandon the builder.
- Forward checksum verification to the external reader while retaining compatibility with implementations that do not override it.
- Add a standalone test implementation shared by basic and full modes. It uses a simple length-prefixed file format with a RocksDB properties block and checksum, avoiding dependencies on native table implementations.
## Crash testing
Add `use_full_external_table` support to `db_stress` and a dedicated blackbox `db_crashtest.py` profile. The profile exercises randomized reads, writes, merges, ingestion, flushes, compactions, snapshots, checksum verification, and crash recovery.
The profile disables unsupported combinations, including range deletions, user timestamps, sequence-number time tracking, backward scans, multiple column families, backup reconstruction, remote compaction, dynamic table-factory replacement, randomized aggregate/all-table property probes, and global-sequence-number rewriting during ingestion.
Because the simple reader materializes each open table in memory, the profile limits open files and generated file sizes to bound total database memory usage.
Reviewed By: xingbowang
Differential Revision: D120637416
fbshipit-source-id: aa2cbd06999e4325a234210d9c45987addd37550
