a38cd3a439
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/15331 ## Motivation The parent diff already implements cross-DB flushable-memory accounting and a background sorter that ranks DBs outside the write path. This change does not alter how the best candidate is selected. The remaining liveness problem occurs after the sorter selects a DB and hands it off for execution. Previously, the coordinator tracked one remote handoff as pending and waited for that DB to report completion before advancing. A selected DB could therefore monopolize coordinator progress if its job waited in Env::LOW, blocked on the DB mutex, waited behind an existing write leader or write stall, or otherwise failed to complete promptly. Other ranked DBs could be ready to reclaim memory but would not be attempted while the first handoff remained pending. ## Solution Make cross-DB flush handoff execution bounded and bypassable. Cross-DB attempts now run on a small four-worker executor owned by the WriteBufferManager instead of consuming Env::LOW compaction capacity. The executor is created only for kFlushLargestAcrossDBs and its workers are retired when that policy is disabled. The coordinator distinguishes queued and executing handoffs. A candidate lease begins when an executor worker starts the callback, rather than while the callback is merely waiting in the executor queue. If an executing handoff exceeds its lease, the coordinator releases ownership of that attempt and advances to the next ranked candidate. The original callback is not cancelled and may still complete, so the executor and outstanding-handoff count provide a strict bound on concurrent stuck work. Each registration has a per-DB outstanding-job reservation, preventing the sorter from dispatching the same DB again while an earlier callback remains alive. The coordinator also records the identity of the active candidate, so late completion from an expired attempt cannot clear or corrupt a newer candidate lease. DB mutex acquisition, write-thread joining, memtable switching, and flush dispatch continue to happen outside registry and coordinator locks. The executor invokes the existing BackgroundCallWBMFlush path directly; actual immutable-memtable flushing still uses RocksDB normal flush infrastructure. ## Compatibility This behavior is limited to kFlushLargestAcrossDBs. Existing policies retain their previous local flush behavior, and no WBM executor threads are started for them. ## Validation Unit coverage exercises lease expiry, advancement to the next candidate, late completion, duplicate suppression, bounded worker capacity, policy changes with blocked callbacks, shutdown, and reranking after progress. The end-to-end stack was also compared against the original local policy in one matched memory-pressure run with 1,000 DBs and 16 writers. The complete largest-across-DBs policy delivered 61.0% higher throughput, 55.5% fewer WBM flushes, 37.9% lower mean latency, 47.9% lower P99 latency, 5.17x larger average flushes, and 62.8% lower peak RSS. These measurements validate the full policy stack rather than isolating the performance contribution of this liveness follow-up. Reviewed By: xingbowang Differential Revision: D123461329 fbshipit-source-id: 4ad86102c2a43beb469f7672d722674dfbda9871
User Documentation for rocksdb.org
This directory will contain the user and feature documentation for RocksDB. The documentation will be hosted on GitHub pages.
Contributing
See CONTRIBUTING.md for details on how to add or modify content.
Run the Site Locally
The requirements for running a GitHub pages site locally is described in GitHub help. The steps below summarize these steps.
If you have run the site before, you can start with step 1 and then move on to step 5.
-
Ensure that you are in the
/docsdirectory in your local RocksDB clone (i.e., the same directory where thisREADME.mdexists). The below RubyGems commands, etc. must be run from there. -
Make sure you have Ruby and RubyGems installed.
Ruby >= 2.2 is required for the gems. On the latest versions of Mac OS X, Ruby 2.0 is the default. Use
brew install ruby(or your preferred upgrade mechanism) to install a newer version of Ruby for your Mac OS X system. -
Make sure you have Bundler installed.
# may require sudo gem install bundler -
Install the project's dependencies
# run this in the 'docs' directory bundle installIf you get an error when installing
nokogiri, you may be running into the problem described in this nokogiri issue. You can eitherbrew uninstall xz(and thenbrew install xzafter the bundle is installed) orxcode-select --install(although this may not work if you have already installed command line tools). -
Run Jekyll's server.
- On first runs or for structural changes to the documentation (e.g., new sidebar menu item), do a full build.
bundle exec jekyll serve- For content changes only, you can use
--incrementalfor faster builds.
bundle exec jekyll serve --incrementalWe use
bundle execinstead of running straightjekyllbecausebundle execwill always use the version of Jekyll from ourGemfile. Just runningjekyllwill use the system version and may not necessarily be compatible.- To run using an actual IP address, you can use
--host=0.0.0.0
bundle exec jekyll serve --host=0.0.0.0This will allow you to use the IP address associated with your machine in the URL. That way you could share it with other people.
e.g., on a Mac, you can your IP address with something like
ifconfig | grep "inet " | grep -v 127.0.0.1. -
Either of commands in the previous step will serve up the site on your local device at http://127.0.0.1:4000/ or http://localhost:4000.
Updating the Bundle
The site depends on Github Pages and the installed bundle is based on the github-pages gem.
Occasionally that gem might get updated with new or changed functionality. If that is the case,
you can run:
bundle update
to get the latest packages for the installation.