Josh Kang e292aa95b3 Allow skipping lmax index and filter block prefetches for external file ingestion ...

Summary:
Bulk external file ingestion can spend significant commit time prefetching index/filter blocks for large Lmax files. This adds `IngestExternalFileOptions::prefetch_lmax_index_and_filter_blocks` (default true) so bulk-load callers can defer that metadata prefetch when the cache doesn't need immediate warming. The option threads through `FileMetaData::skip_index_and_filter_blocks_prefetch` to drive the existing `LoadTableHandlers()` prefetch logic. Benchmarks show ~40% commit latency reduction (3459 → 2112 μs) and ~55% lower IO time (2560 → 1145 μs) when disabled.

Benchmarks:

```
db_bench --benchmarks=ingestexternalfile --num=2200000 --ingest_external_file_num_batches=1 --ingest_external_file_batch_size=1 --ingest_external_file_use_file_info=true --ingest_external_file_fill_cache=true --cache_index_and_filter_blocks=true --bloom_bits=10 --statistics=true --stats_level=3 --ingest_external_file_prefetch_lmax_index_and_filter_blocks=<true|false>
```

| `ingest_external_file_prefetch_lmax_index_and_filter_blocks` | `rocksdb.ingest.external.file.run.micros` | `rocksdb.table.open.io.micros` | index/filter cache adds | last-level read bytes |
| `true` | `3459 us` | `2560 us` | `1 / 1` | `3551559` |
| `false` | `2112 us` | `1145 us` | `0 / 0` | `1333` |

Reviewed By: xingbowang

Differential Revision: D108678511

fbshipit-source-id: e8951e3aad36cc0accffb33ecc3cc1b4aeb89459

2026-06-16 11:20:07 -07:00

..

advisor

Deprecate random access max buffer size references - take #2 (#13288)

2025-01-10 15:32:38 -08:00

block_cache_analyzer

Remove obsolete compression code and some .h->.cc movement (#14325)

2026-02-13 11:18:05 -08:00

dump

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

analyze_txn_stress_test.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

auto_sanity_test.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

backup_db.sh

Revamp check_format_compatible.sh (#8012)

2021-03-02 11:42:27 -08:00

benchmark_ci.py

internal_repo_rocksdb

2024-10-14 03:01:20 -07:00

benchmark_compare.sh

Fix file modes (#10815)

2022-10-13 09:00:37 -07:00

benchmark_leveldb.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

benchmark.sh

optimize file size statistics in benchmark script (#12363)

2024-02-21 15:45:18 -08:00

blob_dump.cc

Restore blob_dump_tool compression support (partially revert PR #14266) (#14382)

2026-02-24 14:10:13 -08:00

check_all_python.py

internal_repo_rocksdb

2024-10-14 03:01:20 -07:00

check_format_compatible.sh

Update version to 11.5.0 for 11.4 release (#14820)

2026-06-05 23:09:42 -07:00

CMakeLists.txt

Mark dependencies as PRIVATE and fix missing dependencies in tools. (#6790)

2020-05-12 21:07:55 -07:00

compact_db.sh

Fix format compatibility issues, extend test (#14323)

2026-02-13 09:18:40 -08:00

db_bench_tool_test.cc

Make file opens multi-threaded for commit external sst file ingestion (#14853)

2026-06-15 14:45:29 -07:00

db_bench_tool.cc

Allow skipping lmax index and filter block prefetches for external file ingestion

2026-06-16 11:20:07 -07:00

db_bench.cc

Add (& fix) some simple source code checks (#8821)

2021-09-07 21:19:27 -07:00

db_crashtest_test.py

Add multi-DB stress testing support (--num_dbs flag) (#14749)

2026-05-27 11:53:11 -07:00

db_crashtest.py

Pass file metadata to IngestExternalFile to improve ingestion latency (#14837)

2026-06-15 11:14:42 -07:00

db_repl_stress.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

db_sanity_test.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

dbench_monitor

Fix /bin/bash shebangs

2017-08-03 15:56:46 -07:00

Dockerfile

adding docker build script and dockerfile

2015-05-22 16:03:39 -07:00

generate_random_db.sh

Add remote compaction format compatibility test to check_format_compatible.sh (#14798)

2026-06-03 14:03:47 -07:00

gtest_parallel_repro.py

Add parallel gtest flake reproduction runner (#14827)

2026-06-12 09:42:58 -07:00

ingest_external_sst.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

io_tracer_parser_test.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

io_tracer_parser_tool.cc

Remove RocksDB LITE (#11147)

2023-01-27 13:14:19 -08:00

io_tracer_parser_tool.h

Remove RocksDB LITE (#11147)

2023-01-27 13:14:19 -08:00

io_tracer_parser.cc

Remove RocksDB LITE (#11147)

2023-01-27 13:14:19 -08:00

ldb_cmd_impl.h

Add remote compaction format compatibility test to check_format_compatible.sh (#14798)

2026-06-03 14:03:47 -07:00

ldb_cmd_test.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

ldb_cmd.cc

Add remote compaction format compatibility test to check_format_compatible.sh (#14798)

2026-06-03 14:03:47 -07:00

ldb_test.py

Restore blob_dump_tool compression support (partially revert PR #14266) (#14382)

2026-02-24 14:10:13 -08:00

ldb_tool.cc

Add remote compaction format compatibility test to check_format_compatible.sh (#14798)

2026-06-03 14:03:47 -07:00

ldb.cc

Use/endorse (Auto)HyperClockCache by default over LRUCache (#13964)

2025-09-18 13:27:51 -07:00

pflag

Fix /bin/bash shebangs

2017-08-03 15:56:46 -07:00

reduce_levels_test.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

regression_test.sh

Clean up failed regression test workdirs (#14751)

2026-05-18 16:24:56 -07:00

restore_db.sh

Revamp check_format_compatible.sh (#8012)

2021-03-02 11:42:27 -08:00

rocksdb_dump_test.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

run_blob_bench.sh

add exe and script path check (#11621)

2023-07-19 12:05:24 -07:00

run_clang_tidy.py

Start development 11.4 (#14747)

2026-05-18 13:35:11 -07:00

run_flash_bench.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

run_leveldb.sh

Add copyright headers per FB open-source checkup tool. (#5199)

2019-04-18 10:55:01 -07:00

sample-dump.dmp

First version of rocksdb_dump and rocksdb_undump.

2015-06-19 16:24:36 -07:00

simulated_hybrid_file_system.cc

Standardize on clang-format version 18 (#13233)

2024-12-19 10:58:40 -08:00

simulated_hybrid_file_system.h

Standardize on clang-format version 18 (#13233)

2024-12-19 10:58:40 -08:00

sst_dump_test.cc

Disable parallel compression for fast built-in compressors (#14841)

2026-06-10 12:58:47 -07:00

sst_dump_tool.cc

Disable parallel compression for fast built-in compressors (#14841)

2026-06-10 12:58:47 -07:00

sst_dump.cc

Remove RocksDB LITE (#11147)

2023-01-27 13:14:19 -08:00

tool_hooks.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

trace_analyzer_test.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00

trace_analyzer_tool.cc

Fix range deletion crash test errors (#14573)

2026-04-06 09:05:42 -07:00

trace_analyzer_tool.h

Fix range deletion crash test errors (#14573)

2026-04-06 09:05:42 -07:00

trace_analyzer.cc

Remove RocksDB LITE (#11147)

2023-01-27 13:14:19 -08:00

tsan_suppressions.txt

env: suppress liburing TSAN false positives (#14710)

2026-05-07 16:40:56 -07:00

verify_random_db.sh

Fix ldb dump swallowing errors, but ldb scan for compatibility test (#14422)

2026-03-05 09:41:58 -08:00

write_external_sst.sh

Revamp check_format_compatible.sh (#8012)

2021-03-02 11:42:27 -08:00

write_stress_runner.py

internal_repo_rocksdb

2024-10-14 03:01:20 -07:00

write_stress.cc

Remove deprecated DB::Open raw pointer variants (and more) (#14335)

2026-02-17 23:33:39 -08:00