Summary:
To start, I wanted to remove the unnecessary new_options parameter of `InstallSuperVersionAndScheduleWork()`. Passing it something other than the latest mutable options would be inconsistent/outdated. There was even a comment "Use latest MutableCFOptions" on a place that was using the saved options in effect for the compaction.
On investigation, this fixes an undiagnosed but longstanding serious bug in SetOptions() where the new settings can be reverted if a flush or compaction started before the SetOptions() finishes after. Fix confirmed with new unit test in db_test.cc.
I also got tired of seeing the cumbersome usage of pointer rather than const reference for related options accesses, so there's kind of a large (but trivial) refactoring tied in here as well. (Sorry for combining them; wasn't planning a major bug fix)
Intended follow-up: Clarify/simplify the crazy calling conventions of LogAndApply, and remove some unnecessary copying of MutableCFOptions (see new FIXMEs)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13294
Test Plan: test for bug fix, confirmed fails on main and at least as far back as version 8.10. Plus existing tests and CI
Reviewed By: mszeszko-meta
Differential Revision: D68141563
Pulled By: pdillinger
fbshipit-source-id: f6c3290145afa06cc2fe8b485a5de17560a5deea
b3333587eb