Summary:
The non-ASCII character check in check-sources.sh used git grep -P (Perl regex), which requires git compiled with PCRE support. On systems without it, the command fails with exit code 128, which is != 1 (no match), so the check always reported a violation -- effectively dead.
Even in CI where git has PCRE2 support, the check was silently broken: git grep -P uses PCRE2 in UTF mode by default, which interprets [\x80-\xFF] as a Unicode codepoint range (U+0080 to U+00FF). Characters like em-dash (U+2014), arrows (U+2192), and math symbols (U+2248, etc.) fall outside that range and were not detected. Only Latin-1 Supplement characters (U+0080-U+00FF) would have been caught.
Replace with LC_ALL=C git grep using bash $'[\x80-\xff]' literal byte range, which works with basic regex in the C locale, and replace all non-ASCII characters in non-excluded source files:
- em-dash to --
- arrow to ->
- math symbols to ASCII equivalents (~=, <=, >=)
- box-drawing characters to ASCII art
Also exclude .github/ from the check, as scripts there can use non-ascii without disrupting RocksDB builds on non-UTF-8 systems.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14729
Test Plan: manual / CI (make check-sources passes clean)
Reviewed By: hx235
Differential Revision: D104692574
Pulled By: pdillinger
fbshipit-source-id: 1d884c21056dcd83558b825a04b867f1c08e3f45
Summary:
CONTEXT: MyRocks and other third-party users currently need to understand deep RocksDB internals to sort unsorted data into SST files. The existing pattern requires: opening a full DB instance, configuring VectorRepFactory with universal compaction, disabling auto-compaction, writing unsorted data, manually triggering CompactRange with kForceOptimized, collecting output via GetColumnFamilyMetaData, and finally ingesting via IngestExternalFile with allow_db_generated_files. This is error-prone and requires ~30 lines of RocksDB configuration knowledge.
WHAT: This adds a new utility class, SortedRunBuilder, that wraps all of the above into a simple Create/Add/Finish API. Callers feed in unsorted key-value pairs (in any order, from any number of threads) and receive sorted SST files with seqno=0 ready for ingestion -- or can iterate sorted output directly.
The name SortedRunBuilder was chosen over alternatives like UnsortedSstFileWriter because this utility targets a broad audience: anyone wanting to use RocksDB as an external sort engine, not just users migrating from SstFileWriter. That said, this explicitly removes the sorted-input requirement that SstFileWriter enforces -- callers no longer need to pre-sort their data before writing SST files.
KEY DESIGN DECISIONS:
- Zero changes to core RocksDB internals. This is a pure utility layer that exclusively calls existing public APIs:
* DB::Open(), DB::Put(), DB::Write(), DB::Flush(), DB::CompactRange(), DB::GetColumnFamilyMetaData(), DB::NewIterator(), DestroyDB()
* VectorRepFactory (sort-on-flush memtable)
* BottommostLevelCompaction::kForceOptimized (zero seqnos)
- No modifications to: compaction logic, memtable implementation, SST file format, ingestion logic, or DB open/close paths
- All new code lives in utilities/sorted_run_builder/ and the public header include/rocksdb/utilities/sorted_run_builder.h
- Build file changes are limited to registering the new source/test files
API SURFACE:
SortedRunBuilderOptions opts; opts.temp_dir = "/tmp/sort_work"; std::unique_ptr<SortedRunBuilder> builder; SortedRunBuilder::Create(opts, &builder);
builder->Add(key, value); // any order, thread-safe
builder->AddBatch(&batch); // WriteBatch for throughput
builder->Finish(); // flush + compact + collect
builder->GetOutputFiles(); // sorted SSTs for ingestion
builder->NewIterator(ro); // iterate sorted output
FILES CHANGED:
New: include/rocksdb/utilities/sorted_run_builder.h (public header) New: utilities/sorted_run_builder/sorted_run_builder.cc (implementation) New: utilities/sorted_run_builder/sorted_run_builder_test.cc (12 tests) New: docs/plans/sorted_run_builder_plan.md (design plan) New: docs/plans/sorted_run_builder_usage_guide.md (usage guide) Modified: src.mk, CMakeLists.txt, Makefile (register new files only)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14499
Test Plan:
$ make clean && make -j$(nproc) sorted_run_builder_test $ ./sorted_run_builder_test [==========] Running 12 tests from 1 test case. [ PASSED ] 12 tests. (688 ms total)
Tests cover: basic sort correctness, empty builder, WriteBatch path, concurrent multi-threaded writes, ingestion into a target DB, large random dataset (10K keys), entry/size counters, error cases (Add after Finish, iterator before Finish, empty temp_dir), cleanup verification, and duplicate key handling.
Reviewed By: xingbowang
Differential Revision: D98935972
Pulled By: dannyhchen
fbshipit-source-id: e3bb7f1ea5f6004aab697e4da667fa21292ca250
