0561153b73
Summary: Motivation A WriteBufferManager can be shared by many RocksDB instances and column families. Under the existing policy, the DB whose writer first observes memory pressure flushes its own oldest mutable memtable. That DB might own only a small fraction of the reclaimable memory, so workloads with many DBs or CFs can generate many small flushes while the largest memory consumer remains untouched. Scanning every registered DB while holding process-wide or DB mutexes would improve selection accuracy but would serialize unrelated write paths and is not viable for deployments with thousands of DBs. Previous behavior kFlushOldest was the only behavior: the write-group leader that noticed WBM pressure synchronously selected the oldest eligible mutable memtable in its own DB. The WBM had no per-DB view of reclaimable memory and could not direct reclamation to another DB sharing the same budget. New behavior This change adds configurable WriteBufferFlushPolicy modes: - kFlushOldest preserves the existing behavior and remains the default. - kFlushLargest selects the largest eligible flushable column family in the writing DB. - kFlushLargestAcrossDBs selects the DB advertising the most reclaimable memory across all DBs sharing the WBM, then flushes its largest eligible CF. Atomic-flush DBs advertise and flush the aggregate eligible memory across their CFs. High-level implementation Each writable DB registers a lightweight FlushInitiator with the shared WBM. Per-DB accounting publishes two approximate values using atomics: total mutable memory and the largest flushable CF contribution, including eligible immutable memtables waiting for flush. Allocator reporting is batched, so the allocation hot path avoids registry scans and DB mutexes. A dedicated WBM coordinator thread maintains an approximate ranked snapshot of registered DBs. Refreshes are driven by accounting changes, threshold crossings, and periodic pressure checks. Ranking and O(number of DBs) work happen on that coordinator thread; writers only update counters and signal the coordinator. Registration lifetime is protected independently so DB close can safely drain an in-flight handoff. Threading and flush handoff The coordinator reserves and pins one ranked candidate at a time, without holding a registry lock while invoking the DB. The target DB uses TryLock on its DB mutex and rejects the handoff if it is closing, paused, stopped, stalled, already handling a WBM job, or no longer has flushable memory. Rejected candidates do not block consideration of the next DB. An accepted handoff queues one LOW-priority WBM job. The LOW worker joins the target DB write thread and revalidates eligibility before switching the memtable. The resulting ordinary flush is then scheduled through RocksDB's existing flush machinery on the HIGH-priority flush pool. Keeping the write-thread join off the requesting writer avoids making one DB's write path wait on another DB's mutex or write queue. Only one manager-wide remote handoff is active at a time. Completion or cancellation clears the handoff and requests a rerank. If memory reaches the hard limit while remote coordination is delayed, a rate-limited local fallback lets a writer seal a local memtable to preserve forward progress. Write stalls, shutdown, WaitForCompact, policy changes, and DB deregistration explicitly drain or reject outstanding handoffs. Compatibility The legacy constructor and kFlushOldest behavior remain unchanged unless a new policy is explicitly selected. There are no on-disk format changes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/15047 Reviewed By: xingbowang Differential Revision: D112396106 fbshipit-source-id: 3049adc7034dd4e00d74d7d86e1d19004292d299
User Documentation for rocksdb.org
This directory will contain the user and feature documentation for RocksDB. The documentation will be hosted on GitHub pages.
Contributing
See CONTRIBUTING.md for details on how to add or modify content.
Run the Site Locally
The requirements for running a GitHub pages site locally is described in GitHub help. The steps below summarize these steps.
If you have run the site before, you can start with step 1 and then move on to step 5.
-
Ensure that you are in the
/docsdirectory in your local RocksDB clone (i.e., the same directory where thisREADME.mdexists). The below RubyGems commands, etc. must be run from there. -
Make sure you have Ruby and RubyGems installed.
Ruby >= 2.2 is required for the gems. On the latest versions of Mac OS X, Ruby 2.0 is the default. Use
brew install ruby(or your preferred upgrade mechanism) to install a newer version of Ruby for your Mac OS X system. -
Make sure you have Bundler installed.
# may require sudo gem install bundler -
Install the project's dependencies
# run this in the 'docs' directory bundle installIf you get an error when installing
nokogiri, you may be running into the problem described in this nokogiri issue. You can eitherbrew uninstall xz(and thenbrew install xzafter the bundle is installed) orxcode-select --install(although this may not work if you have already installed command line tools). -
Run Jekyll's server.
- On first runs or for structural changes to the documentation (e.g., new sidebar menu item), do a full build.
bundle exec jekyll serve- For content changes only, you can use
--incrementalfor faster builds.
bundle exec jekyll serve --incrementalWe use
bundle execinstead of running straightjekyllbecausebundle execwill always use the version of Jekyll from ourGemfile. Just runningjekyllwill use the system version and may not necessarily be compatible.- To run using an actual IP address, you can use
--host=0.0.0.0
bundle exec jekyll serve --host=0.0.0.0This will allow you to use the IP address associated with your machine in the URL. That way you could share it with other people.
e.g., on a Mac, you can your IP address with something like
ifconfig | grep "inet " | grep -v 127.0.0.1. -
Either of commands in the previous step will serve up the site on your local device at http://127.0.0.1:4000/ or http://localhost:4000.
Updating the Bundle
The site depends on Github Pages and the installed bundle is based on the github-pages gem.
Occasionally that gem might get updated with new or changed functionality. If that is the case,
you can run:
bundle update
to get the latest packages for the installation.