mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
81ea677072
Summary:
Wide-column read paths that materialize blob-referenced columns used to re-serialize the entire entity into a fresh contiguous buffer, copying the (potentially large) resolved column values byte-for-byte. This was forced by PinnableWideColumns holding a single backing buffer with all column Slices pointing into it.
PinnableWideColumns now holds its columns as zero-copy Slices into one or more stable backing buffers, kept in a std::forward_list<PinnableSlice> whose nodes never relocate on move, move-assignment, or splice. Resolved blob columns can be spliced in as additional backing buffers instead of re-serializing the entity.
Using address-stable nodes also cleans up the old Slice-invalidation handling. The previous single-buffer representation had to cope with a self-pinned PinnableSlice whose small (SSO) std::string relocates when the PinnableSlice is moved: on every move it detected the relocated backing pointer and re-derived the column Slices by re-parsing. With forward_list nodes that never move, the column Slices stay valid across move / move-assignment / splice, so that handling is gone and the move operations are now the defaulted member-wise moves.
The multi-buffer plumbing is internal only: a new PinnableWideColumnsHelper (a friend) performs the splice and the little bit of private-member access the read paths need, so there is no new public API surface and the previous `friend class DBImpl` is gone. The public view is unchanged: columns() still returns a single, in-order set.
Read paths converted to resolve without re-serializing:
- GetContext::SaveWideColumnEntityToColumns (eager GetEntity SST reads, separate-blob-file references)
- DBImpl::ResolveDirectWriteWideColumns (direct-write blob references)
- DBImpl::MaybeResolveMemtableBlobValue (blob-backed memtable reads)
Merge output still serializes (the MergeOperator API consumes/produces a serialized entity). Same-file ("embedded") blob columns are resolved -- and still re-serialized -- earlier in the block-based table reader (MaybeResolveEmbeddedValue / ResolveEmbeddedValueForGet) before the entity reaches GetContext, so the GetEntity path above only avoids re-serialization for separate-blob-file references. Compaction's separate entity machinery is unchanged.
API/accounting: deprecated PinnableWideColumns::serialized_size() (comment-only, per repo convention) in favor of the new payload_size() (sum of column name + value sizes; equals the value size for a plain value). Read statistics for entities (BYTES_READ, BYTES_PER_READ, per-key GetEntity/MultiGetEntity byte counts) and the ReadOptions::value_size_soft_limit threshold now measure entities by payload_size() instead of the serialized size. This drops framing bytes (entity measurements are slightly smaller), thus making plain-value and wide-column accounting consistent and reducing unnecessary exposure to implementation details.
Serialization cleanup enabled by the above:
- Consolidated Deserialize/DeserializeV2 into a single Deserialize(const Slice&, WideColumns&, blob_columns*) plus a DeserializeSimple alias for the no-blob-references case; both take const Slice&, require the input to be exactly one serialized entity, and reject any trailing data uniformly (V1 previously ignored trailing bytes). An unexpected blob reference where a fully-resolved entity is required now returns Corruption.
- ResolveEntityBlobColumns (serialized output, used only by merge) is now a thin wrapper over the new multi-buffer resolver plus Serialize, removing the duplicated fetch loop and the SerializeResolvedEntity helper.
- Consistently use `const Slice& input` rather than `Slice& input` for wide column functions expecting an input slice with no trailing data.
Design/polish from human review:
* The initial plan called for more complex handling around a vector<PinnableSlice> backing store (unique_ptr indirection and re-deriving the column Slices on move to cope with reallocation and SSO relocation); switching the backing store to a std::forward_list of address-stable nodes was a human suggestion that removed that complexity and the move-time Slice re-derivation entirely.
* Keep resolve plumbing off the public API via PinnableWideColumnsHelper and removing the DBImpl friend
* Various pieces of cleanup and polish (more left to do)
* The eager-GetEntity optimization
* The memtable inline-copy NOTE/TODO.
Intended follow-up:
* Resolve FIXME about pre-existing swallowed non-OK status in db_impl.cc
* Resolve TODO about value_size_soft_limit in db_impl.cc
* Extend this optimization to SST embedded-blob references
* Error-status/API cleanup: classify an unrecognized wide-column version as Corruption (not NotSupported) at all entry points; replace the NotSupported sentinel from GetValueOfDefaultColumn with an explicit is_blob_reference out-param (removing GetValueOfDefaultColumnResolvingBlobs); and centralize the "blob-backed value without a blob fetcher" Corruption into the resolvers (resolving the ResolveEntityForMerge FIXME) rather than repeating it at each caller
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14943
Test Plan:
Extended db/wide/wide_column_serialization_test.cc with PinnableWideColumns unit tests: set/payload_size for plain, V1, and resolved entities; move and move-assign of a multi-buffer result preserve column data pointers (the node-stability guarantee); Reset; ResolveColumns splices blob buffers in zero-copy (resolved values' data() point into the spliced-in buffers, not a re-serialized entity); serialized_size() == SerializedSizeV1(columns()); and Deserialize rejects trailing data after a serialized entity for both V1 and V2 layouts.
Built and ran (debug): wide_column_serialization_test, db_wide_basic_test, db_wide_blob_direct_write_test, db_blob_basic_test, compaction_iterator_test, db_merge_operator_test, db_iterator_test, udt_util_test, write_batch_with_index_test -- all pass. librocksdb and db_stress build. make format-auto and build_tools/check-sources.sh are clean.
Reviewed By: xingbowang
Differential Revision: D112212266
Pulled By: pdillinger
fbshipit-source-id: 13b89e2bb7d8371f4c1ff18b234a549bea9c9e77
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.