1
0
mirror of https://github.com/facebook/rocksdb.git synced 2026-10-23 08:09:02 +03:00
Files
rocksdb/db/coalescing_iterator.cc
Xingbo Wang 3070f73e97 Wide-column blob separation: lazy resolution through read, compaction, and write paths (#14386)
Summary:
Wide-column blob separation: lazy resolution through read, compaction, and write paths

Extend blob direct write to support wide-column entities (PutEntity), and add
lazy blob resolution for wide-column values across all read and compaction paths.

**Write path -- PutEntity blob separation:**
- BlobWriteBatchTransformer::PutEntityCF now extracts large column values
  (>= min_blob_size) to blob files and serializes V2 entities with BlobIndex
  references, matching the existing Put behavior.
- Add MaybePreprocessWideColumns() static helper to share blob extraction
  logic between the WriteBatch transformer and the new PutEntity fast path.
- Add PutEntityFastPath() in DBImpl that preprocesses columns (sort, blob
  extract, serialize) before calling WriteImpl, skipping the redundant
  WriteBatch transformation pass. Trace batch preserves the original columns.

**Read path -- blob resolution for Get/MultiGet/Iterator:**
- GetContext::SaveValue resolves V2 entity blob columns eagerly: for
  value (Get), resolves the default column's blob reference; for columns
  (GetEntity), resolves all blob columns and re-serializes as V1.
- DBIter::SetValueAndColumnsFromEntity detects V2 entities, deserializes
  with DeserializeV2, and eagerly resolves all blob columns via a new
  ReadPathBlobResolver. Resolved values are cached in the resolver and
  wide_columns_ Slices point into the cache, avoiding copies.
- Add ReadPathBlobResolver (new file) -- on-demand blob fetcher for the
  read path with per-column caching, used by both DBIter and GetContext.
- BlobFetcher gains allow_write_path_fallback to read from in-flight
  direct-write blob files not yet visible through Version (pre-flush reads).
- Memtable lookups for Get(key) on V2 entities with a blob default column
  now return the blob index with is_blob_index=true, triggering the
  existing BDW resolution in MaybeResolveWritePathValue.
- MaybeResolveWritePathValue (renamed from MaybeResolveDirectWriteBlobIndex)
  now also resolves V2 entity blob columns for GetEntity/MultiGetEntity,
  re-serializing as V1 after resolution.

**Compaction path -- filter, GC, and extraction:**
- CompactionIterator::InvokeFilterIfNeeded handles V2 entities: FilterV3
  gets eagerly-resolved column values for backward compatibility; FilterV4
  gets a CompactionBlobResolver for lazy on-demand resolution.
- Add CompactionFilter::FilterV4 with WideColumnBlobResolver* parameter
  and SupportsFilterV4() opt-in. Default delegates to FilterV3.
- CompactionBlobResolver (new class) implements WideColumnBlobResolver
  for the compaction path with stats tracking.
- ExtractLargeColumnValuesIfNeeded extracts inline columns to blob files
  during compaction (entities without existing blob columns only).
- GarbageCollectEntityBlobsIfNeeded relocates blob values from old blob
  files to new ones during compaction GC, with helpers FetchBlobsNeedingGC,
  RelocateBlobValues, and SerializeEntityAfterGC.
- PrepareOutput unified entity deserialization: single DeserializeV2 call
  reused by both filter and GC/extraction paths via entity_deserialized_
  flag, avoiding redundant parsing.

**Merge path -- V2 entity base value resolution:**
- MergeHelper::MergeUntil, GetContext::MergeWithWideColumnBaseValue, and
  DBIter::MergeWithWideColumnBaseValue resolve V2 blob columns before
  calling TimedFullMerge, using ResolveEntityForMerge.

**Blob garbage accounting:**
- BlobGarbageMeter tracks blob file in/out flow for V2 entity blob
  columns via ForEachBlobFileNumber, used for accurate GC decisions.
- FileMetaData::UpdateBoundaries tracks oldest_blob_file_number for
  V2 entities, ensuring blob files referenced by entities are not
  prematurely deleted.

**Serialization improvements:**
- WideColumnSerialization::SerializeV2Impl allocates serialized_blob_indices
  only for actual blob columns (not all columns) and uses autovector for
  name/value sizes.
- Add ForEachBlobFileNumber for lightweight blob file number extraction
  without full deserialization.
- Add ResolveEntityForMerge helper for merge-path resolution.
- Add section-size validation in DeserializeV2Impl.
- Add empty blob index and column type validation.
- blob_column_resolver_util.h -- shared helpers (FindBlobColumn, FindInCache,
  CacheInlinedBlob) used by both ReadPathBlobResolver and CompactionBlobResolver.

**Testing:**
- db_blob_direct_write_test: end-to-end PutEntity with BDW before/after flush,
  verifying Get, GetEntity, MultiGetEntity, and Iterator.
- db_blob_index_test: ~1550 lines covering V2 entity blob resolution through
  Get, GetEntity, MultiGet, Iterator, compaction filter (V3 compat and V4 lazy),
  merge with blob base, and compaction GC/extraction.
- compaction_iterator_test: ~950 lines testing entity blob GC, extraction,
  filter interaction, and combined GC+filter scenarios.
- db_wide_basic_test: ~1200 lines for wide-column lazy blob resolution through
  all read paths plus compaction round-trips.
- db_open_with_config_test: ~450 lines for BDW entity config validation.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/14386

Reviewed By: anand1976

Differential Revision: D99739701

Pulled By: xingbowang

fbshipit-source-id: 6badd89b577f3054802eaaa654738468efb9dbdb
2026-04-08 18:58:40 -07:00

58 lines
1.8 KiB
C++

// Copyright (c) Meta Platforms, Inc. and affiliates.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
#include "db/coalescing_iterator.h"
#include "db/wide/wide_columns_helper.h"
namespace ROCKSDB_NAMESPACE {
void CoalescingIterator::Coalesce(
const autovector<MultiCfIteratorInfo>& items) {
assert(wide_columns_.empty());
assert(owned_columns_.empty());
MinHeap heap;
for (const auto& item : items) {
assert(item.iterator);
for (auto& column : item.iterator->columns()) {
heap.push(WideColumnWithOrder{&column, item.order});
}
}
if (heap.empty()) {
return;
}
// Own the coalesced bytes so the result stays valid even if a child iterator
// refreshes or reuses its internal buffers.
owned_columns_.reserve(heap.size());
wide_columns_.reserve(heap.size());
auto add_column = [this](const WideColumn& column) {
owned_columns_.emplace_back(column.name().ToString(),
column.value().ToString());
const auto& owned = owned_columns_.back();
wide_columns_.emplace_back(owned.first, owned.second);
};
auto current = heap.top();
heap.pop();
while (!heap.empty()) {
int comparison = current.column->name().compare(heap.top().column->name());
if (comparison < 0) {
add_column(*current.column);
} else if (comparison > 0) {
// Shouldn't reach here.
// Current item in the heap is greater than the top item in the min heap
assert(false);
}
current = heap.top();
heap.pop();
}
add_column(*current.column);
if (WideColumnsHelper::HasDefaultColumn(wide_columns_)) {
value_ = WideColumnsHelper::GetDefaultColumn(wide_columns_);
}
}
} // namespace ROCKSDB_NAMESPACE