mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
fd7399004f
Summary:
Adds an experimental **lazy wide-column read API** that lets callers enumerate a
wide-column entity and read back only the columns (and, in a later phase, the
byte ranges) they actually need, instead of eagerly materializing every
blob-backed column like `GetEntity()` / `MultiGetEntity()`.
New public entry points (`include/rocksdb/db.h`):
- `DB::GetEntityLazy(read_options, cf, key, LazyWideColumns* result)`
- `DB::MultiGetEntityLazy(read_options, cf, num_keys, keys, LazyWideColumnsBatch* result, statuses, sorted_input)`
Both leave a wide-column entity's **inline** columns materialized up front but
its **blob-backed** columns as *unresolved references*, so blobs for columns
that are never pulled are never read.
### Result types (`include/rocksdb/lazy_wide_columns.h`)
- **`LazyWideColumn`** — the lazy analogue of `WideColumn`: `name()`, `index()`,
`inline_value()` (an `OptSlice`; empty ⇒ unresolved reference),
`is_reference()`, and `logical_size()` (`std::optional<uint64_t>`; known for
inline and uncompressed references, empty for compressed ones). All no-I/O.
- **`LazyWideColumns`** — a move-only result that is **indexable and iterable
like a vector** of `LazyWideColumn` (`size()`, `operator[]`, range `begin/end`).
Unlike `PinnableWideColumns`, it may be used after the producing call returns:
it **pins the SuperVersion** (like an iterator) so referenced blob files / SST
readers stay valid for deferred reads.
- **`LazyWideColumnsBatch`** — the `MultiGetEntityLazy` result; vector-like over
its per-key `LazyWideColumns`.
### Resolution
- **`LazyColumnReadRequest { const LazyWideColumn* column; uint64_t offset;
size_t length; bool force_verify; PinnableSlice* result; Status* status; }`** —
a read names a **typed column pointer**, not an untyped index. Because a
column identifies its owning result (and, for a batch, its owning entity),
`MultiResolve` routes each read correctly and **rejects a null or foreign
column** with `InvalidArgument` on that read's per-read status — cross-result
misuse can't be silently mis-indexed.
- **`LazyWideColumns::MultiResolve` / `LazyWideColumnsBatch::MultiResolve`** take
the same request type (pointer+count and `std::vector` sugar overloads); the
batch form resolves reads spanning any entities of the batch in one call.
- **`ResolveColumn(column, result)` / `ResolveColumnRange(column, offset, length,
result)`** — per-column sugar over a one-entry `MultiResolve`.
- A column's blob is fetched **at most once** over the result's lifetime (bytes
are cached in the result), so repeated reads of the same column reuse that
fetch. Resolution honors the `ReadOptions` captured at `GetEntityLazy()` time
(e.g. `read_tier`, `verify_checksums`); the resolve methods intentionally take
no per-resolve `ReadOptions`.
### Where it works
- Primary, **read-only**, and **secondary** DB instances (each `GetImpl`
override forwards the lazy signal and transfers a SuperVersion pin via the
shared `DBImpl::TransferSuperVersionPin` helper).
- Forwarded through `StackableDB` wrappers (e.g. `TransactionDB`) rather than
falling through to the `NotSupported` base default.
- Requires `max_open_files == -1` (immortal table readers) so same-file
("embedded") references stay resolvable lazily; otherwise the call returns
`InvalidArgument`.
## Implementation notes
- Read path: `GetContext` gains a lazy mode that leaves blob references
unresolved and captures the SST's `SameFileBlobReader`; `Version::Get` forwards
the new (defaulted) `lazy_columns_same_file_reader` out-param.
- `ReadPathBlobResolver` performs on-demand, cached blob resolution bound to the
pinned `Version` + decoded blob references.
- Internal identity is by heap-stable `Rep` pointers, so column/result/batch
pointers stay valid across moves with no move-time fix-ups.
## Scope / current limitations
- Resolves **whole columns**; byte ranges are served by slicing the whole column
for now (so `length`/`offset`/`force_verify` don't yet save I/O or skip
checksums). Single column family. Synchronous. `kBlockCacheTier` on the
originating call yields `Incomplete` on a blob-cache miss.
- User-defined timestamps: entities aren't writable under UDT, so `GetEntityLazy`
is at parity with `GetEntity` (same read-timestamp validation; plain values
return a single default column).
- Planned follow-ups: partial (byte-range) blob reads that skip checksum/cache-
fill, cross-key coalescing, and asynchronous execution.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/15012
Test Plan:
- **`db/wide/db_lazy_entity_test.cc` (15 tests, all passing):** enumeration with
no blob I/O; single range and whole-column resolution; `MultiResolve` of many
blobs and mixed ranges/columns (incl. the `std::vector` overload); cross-key
batch resolution; result-outlives-call / pin survives move; unpulled columns
never read; `max_open_files == -1` requirement; `kBlockCacheTier` ⇒
`Incomplete`; foreign/null column ⇒ `InvalidArgument` (single-entity and
batch); UDT parity with `GetEntity`; `StackableDB` forwarding; and read-only
and secondary instances.
- **db_stress:** `GetEntityLazy` / `MultiGetEntityLazy` run as a differential
against eager `GetEntity` / `MultiGetEntity` (gated by `--lazy_entity_read_one_in`,
which only takes effect under `--open_files=-1`), including coverage on
read-only (`--open_read_only_one_in`) and secondary (`--test_secondary`)
instances. Smoke-run (`EXIT=0`, "Verification successful") with integrated blob
files + `--use_put_entity_one_in`.
- `make format-auto` clean; builds `ASSERT_STATUS_CHECKED`-clean.
Reviewed By: xingbowang
Differential Revision: D113792365
Pulled By: pdillinger
fbshipit-source-id: cb73fc845fe0e20ed352d697e552e7b47b2eb9cf
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage
RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)
This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database.
Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples
See the github wiki for more explanation.
The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.
Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.
License
RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.