mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
81ea677072
Summary:
Wide-column read paths that materialize blob-referenced columns used to re-serialize the entire entity into a fresh contiguous buffer, copying the (potentially large) resolved column values byte-for-byte. This was forced by PinnableWideColumns holding a single backing buffer with all column Slices pointing into it.
PinnableWideColumns now holds its columns as zero-copy Slices into one or more stable backing buffers, kept in a std::forward_list<PinnableSlice> whose nodes never relocate on move, move-assignment, or splice. Resolved blob columns can be spliced in as additional backing buffers instead of re-serializing the entity.
Using address-stable nodes also cleans up the old Slice-invalidation handling. The previous single-buffer representation had to cope with a self-pinned PinnableSlice whose small (SSO) std::string relocates when the PinnableSlice is moved: on every move it detected the relocated backing pointer and re-derived the column Slices by re-parsing. With forward_list nodes that never move, the column Slices stay valid across move / move-assignment / splice, so that handling is gone and the move operations are now the defaulted member-wise moves.
The multi-buffer plumbing is internal only: a new PinnableWideColumnsHelper (a friend) performs the splice and the little bit of private-member access the read paths need, so there is no new public API surface and the previous `friend class DBImpl` is gone. The public view is unchanged: columns() still returns a single, in-order set.
Read paths converted to resolve without re-serializing:
- GetContext::SaveWideColumnEntityToColumns (eager GetEntity SST reads, separate-blob-file references)
- DBImpl::ResolveDirectWriteWideColumns (direct-write blob references)
- DBImpl::MaybeResolveMemtableBlobValue (blob-backed memtable reads)
Merge output still serializes (the MergeOperator API consumes/produces a serialized entity). Same-file ("embedded") blob columns are resolved -- and still re-serialized -- earlier in the block-based table reader (MaybeResolveEmbeddedValue / ResolveEmbeddedValueForGet) before the entity reaches GetContext, so the GetEntity path above only avoids re-serialization for separate-blob-file references. Compaction's separate entity machinery is unchanged.
API/accounting: deprecated PinnableWideColumns::serialized_size() (comment-only, per repo convention) in favor of the new payload_size() (sum of column name + value sizes; equals the value size for a plain value). Read statistics for entities (BYTES_READ, BYTES_PER_READ, per-key GetEntity/MultiGetEntity byte counts) and the ReadOptions::value_size_soft_limit threshold now measure entities by payload_size() instead of the serialized size. This drops framing bytes (entity measurements are slightly smaller), thus making plain-value and wide-column accounting consistent and reducing unnecessary exposure to implementation details.
Serialization cleanup enabled by the above:
- Consolidated Deserialize/DeserializeV2 into a single Deserialize(const Slice&, WideColumns&, blob_columns*) plus a DeserializeSimple alias for the no-blob-references case; both take const Slice&, require the input to be exactly one serialized entity, and reject any trailing data uniformly (V1 previously ignored trailing bytes). An unexpected blob reference where a fully-resolved entity is required now returns Corruption.
- ResolveEntityBlobColumns (serialized output, used only by merge) is now a thin wrapper over the new multi-buffer resolver plus Serialize, removing the duplicated fetch loop and the SerializeResolvedEntity helper.
- Consistently use `const Slice& input` rather than `Slice& input` for wide column functions expecting an input slice with no trailing data.
Design/polish from human review:
* The initial plan called for more complex handling around a vector<PinnableSlice> backing store (unique_ptr indirection and re-deriving the column Slices on move to cope with reallocation and SSO relocation); switching the backing store to a std::forward_list of address-stable nodes was a human suggestion that removed that complexity and the move-time Slice re-derivation entirely.
* Keep resolve plumbing off the public API via PinnableWideColumnsHelper and removing the DBImpl friend
* Various pieces of cleanup and polish (more left to do)
* The eager-GetEntity optimization
* The memtable inline-copy NOTE/TODO.
Intended follow-up:
* Resolve FIXME about pre-existing swallowed non-OK status in db_impl.cc
* Resolve TODO about value_size_soft_limit in db_impl.cc
* Extend this optimization to SST embedded-blob references
* Error-status/API cleanup: classify an unrecognized wide-column version as Corruption (not NotSupported) at all entry points; replace the NotSupported sentinel from GetValueOfDefaultColumn with an explicit is_blob_reference out-param (removing GetValueOfDefaultColumnResolvingBlobs); and centralize the "blob-backed value without a blob fetcher" Corruption into the resolvers (resolving the ResolveEntityForMerge FIXME) rather than repeating it at each caller
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14943
Test Plan:
Extended db/wide/wide_column_serialization_test.cc with PinnableWideColumns unit tests: set/payload_size for plain, V1, and resolved entities; move and move-assign of a multi-buffer result preserve column data pointers (the node-stability guarantee); Reset; ResolveColumns splices blob buffers in zero-copy (resolved values' data() point into the spliced-in buffers, not a re-serialized entity); serialized_size() == SerializedSizeV1(columns()); and Deserialize rejects trailing data after a serialized entity for both V1 and V2 layouts.
Built and ran (debug): wide_column_serialization_test, db_wide_basic_test, db_wide_blob_direct_write_test, db_blob_basic_test, compaction_iterator_test, db_merge_operator_test, db_iterator_test, udt_util_test, write_batch_with_index_test -- all pass. librocksdb and db_stress build. make format-auto and build_tools/check-sources.sh are clean.
Reviewed By: xingbowang
Differential Revision: D112212266
Pulled By: pdillinger
fbshipit-source-id: 13b89e2bb7d8371f4c1ff18b234a549bea9c9e77
509 lines
21 KiB
C++
509 lines
21 KiB
C++
// Copyright (c) Meta Platforms, Inc. and affiliates.
|
|
//
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
#include "util/udt_util.h"
|
|
|
|
#include <gtest/gtest.h>
|
|
|
|
#include "db/dbformat.h"
|
|
#include "test_util/testharness.h"
|
|
#include "test_util/testutil.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
namespace {
|
|
static const std::string kTestKeyWithoutTs = "key";
|
|
static const std::string kValuePlaceHolder = "value";
|
|
static const uint64_t kWriteUnixTime = 100;
|
|
} // namespace
|
|
|
|
class HandleTimestampSizeDifferenceTest : public testing::Test {
|
|
public:
|
|
HandleTimestampSizeDifferenceTest() = default;
|
|
|
|
// Test handler used to collect the column family id and user keys contained
|
|
// in a WriteBatch for test verification. And verifies the value part stays
|
|
// the same if it's available.
|
|
class KeyCollector : public WriteBatch::Handler {
|
|
public:
|
|
explicit KeyCollector() = default;
|
|
|
|
~KeyCollector() override = default;
|
|
|
|
Status PutCF(uint32_t cf, const Slice& key, const Slice& value) override {
|
|
if (value.compare(kValuePlaceHolder) != 0) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status TimedPutCF(uint32_t cf, const Slice& key, const Slice& value,
|
|
uint64_t write_unix_time) override {
|
|
if (value.compare(kValuePlaceHolder) != 0) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
if (write_unix_time != kWriteUnixTime) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status PutEntityCF(uint32_t cf, const Slice& key,
|
|
const Slice& entity) override {
|
|
Slice entity_copy = entity;
|
|
WideColumns columns;
|
|
Status s =
|
|
WideColumnSerialization::DeserializeSimple(entity_copy, columns);
|
|
if (!s.ok()) {
|
|
return s;
|
|
}
|
|
if (columns.size() != 1) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
if (columns[0].value().compare(kValuePlaceHolder) != 0) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status DeleteCF(uint32_t cf, const Slice& key) override {
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status SingleDeleteCF(uint32_t cf, const Slice& key) override {
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status DeleteRangeCF(uint32_t cf, const Slice& begin_key,
|
|
const Slice& end_key) override {
|
|
Status status = AddKey(cf, begin_key);
|
|
if (!status.ok()) {
|
|
return status;
|
|
}
|
|
return AddKey(cf, end_key);
|
|
}
|
|
|
|
Status MergeCF(uint32_t cf, const Slice& key, const Slice& value) override {
|
|
if (value.compare(kValuePlaceHolder) != 0) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status PutBlobIndexCF(uint32_t cf, const Slice& key,
|
|
const Slice& value) override {
|
|
if (value.compare(kValuePlaceHolder) != 0) {
|
|
return Status::InvalidArgument();
|
|
}
|
|
return AddKey(cf, key);
|
|
}
|
|
|
|
Status MarkBeginPrepare(bool) override { return Status::OK(); }
|
|
|
|
Status MarkEndPrepare(const Slice&) override { return Status::OK(); }
|
|
|
|
Status MarkRollback(const Slice&) override { return Status::OK(); }
|
|
|
|
Status MarkCommit(const Slice&) override { return Status::OK(); }
|
|
|
|
Status MarkCommitWithTimestamp(const Slice&, const Slice&) override {
|
|
return Status::OK();
|
|
}
|
|
|
|
Status MarkNoop(bool) override { return Status::OK(); }
|
|
|
|
const std::vector<std::pair<uint32_t, const Slice>>& GetKeys() const {
|
|
return keys_;
|
|
}
|
|
|
|
private:
|
|
Status AddKey(uint32_t cf, const Slice& key) {
|
|
keys_.emplace_back(cf, key);
|
|
return Status::OK();
|
|
}
|
|
std::vector<std::pair<uint32_t, const Slice>> keys_;
|
|
};
|
|
|
|
void CreateKey(std::string* key_buf, size_t ts_sz) {
|
|
if (ts_sz > 0) {
|
|
AppendKeyWithMinTimestamp(key_buf, kTestKeyWithoutTs, ts_sz);
|
|
} else {
|
|
key_buf->assign(kTestKeyWithoutTs);
|
|
}
|
|
}
|
|
|
|
void CreateWriteBatch(const UnorderedMap<uint32_t, size_t>& ts_sz_for_batch,
|
|
WriteBatch* batch) {
|
|
for (const auto& [cf_id, ts_sz] : ts_sz_for_batch) {
|
|
std::string key;
|
|
CreateKey(&key, ts_sz);
|
|
ASSERT_OK(WriteBatchInternal::Put(batch, cf_id, key, kValuePlaceHolder));
|
|
ASSERT_OK(WriteBatchInternal::Delete(batch, cf_id, key));
|
|
ASSERT_OK(WriteBatchInternal::SingleDelete(batch, cf_id, key));
|
|
ASSERT_OK(WriteBatchInternal::DeleteRange(batch, cf_id, key, key));
|
|
ASSERT_OK(
|
|
WriteBatchInternal::Merge(batch, cf_id, key, kValuePlaceHolder));
|
|
ASSERT_OK(WriteBatchInternal::PutBlobIndex(batch, cf_id, key,
|
|
kValuePlaceHolder));
|
|
ASSERT_OK(WriteBatchInternal::TimedPut(
|
|
batch, cf_id, key, kValuePlaceHolder, kWriteUnixTime));
|
|
WideColumns columns{{kDefaultWideColumnName, kValuePlaceHolder}};
|
|
ASSERT_OK(WriteBatchInternal::PutEntity(batch, cf_id, key, columns));
|
|
}
|
|
}
|
|
|
|
void CheckSequenceEqual(const WriteBatch& orig_batch,
|
|
const WriteBatch& new_batch) {
|
|
ASSERT_EQ(WriteBatchInternal::Sequence(&orig_batch),
|
|
WriteBatchInternal::Sequence(&new_batch));
|
|
}
|
|
void CheckCountEqual(const WriteBatch& orig_batch,
|
|
const WriteBatch& new_batch) {
|
|
ASSERT_EQ(WriteBatchInternal::Count(&orig_batch),
|
|
WriteBatchInternal::Count(&new_batch));
|
|
}
|
|
|
|
void VerifyKeys(
|
|
const std::vector<std::pair<uint32_t, const Slice>>& keys_with_ts,
|
|
const std::vector<std::pair<uint32_t, const Slice>>& keys_without_ts,
|
|
size_t ts_sz, std::optional<uint32_t> dropped_cf) {
|
|
ASSERT_EQ(keys_with_ts.size(), keys_without_ts.size());
|
|
const std::string kTsMin(ts_sz, static_cast<unsigned char>(0));
|
|
for (size_t i = 0; i < keys_with_ts.size(); i++) {
|
|
// TimestampRecoveryHandler ignores dropped column family and copy it over
|
|
// as is. Check the keys stay the same.
|
|
if (dropped_cf.has_value() &&
|
|
keys_with_ts[i].first == dropped_cf.value()) {
|
|
ASSERT_EQ(keys_with_ts[i].first, keys_without_ts[i].first);
|
|
ASSERT_EQ(keys_with_ts[i].second, keys_without_ts[i].second);
|
|
continue;
|
|
}
|
|
const Slice& key_with_ts = keys_with_ts[i].second;
|
|
const Slice& key_without_ts = keys_without_ts[i].second;
|
|
ASSERT_TRUE(key_with_ts.starts_with(key_without_ts));
|
|
ASSERT_EQ(key_with_ts.size() - key_without_ts.size(), ts_sz);
|
|
ASSERT_TRUE(key_with_ts.ends_with(kTsMin));
|
|
}
|
|
}
|
|
|
|
void CheckContentsWithTimestampStripping(const WriteBatch& orig_batch,
|
|
const WriteBatch& new_batch,
|
|
size_t ts_sz,
|
|
std::optional<uint32_t> dropped_cf) {
|
|
CheckSequenceEqual(orig_batch, new_batch);
|
|
CheckCountEqual(orig_batch, new_batch);
|
|
KeyCollector collector_for_orig_batch;
|
|
ASSERT_OK(orig_batch.Iterate(&collector_for_orig_batch));
|
|
KeyCollector collector_for_new_batch;
|
|
ASSERT_OK(new_batch.Iterate(&collector_for_new_batch));
|
|
VerifyKeys(collector_for_orig_batch.GetKeys(),
|
|
collector_for_new_batch.GetKeys(), ts_sz, dropped_cf);
|
|
}
|
|
|
|
void CheckContentsWithTimestampPadding(const WriteBatch& orig_batch,
|
|
const WriteBatch& new_batch,
|
|
size_t ts_sz) {
|
|
CheckSequenceEqual(orig_batch, new_batch);
|
|
CheckCountEqual(orig_batch, new_batch);
|
|
KeyCollector collector_for_orig_batch;
|
|
ASSERT_OK(orig_batch.Iterate(&collector_for_orig_batch));
|
|
KeyCollector collector_for_new_batch;
|
|
ASSERT_OK(new_batch.Iterate(&collector_for_new_batch));
|
|
VerifyKeys(collector_for_new_batch.GetKeys(),
|
|
collector_for_orig_batch.GetKeys(), ts_sz,
|
|
std::nullopt /* dropped_cf */);
|
|
}
|
|
};
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest, AllColumnFamiliesConsistent) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, sizeof(uint64_t)},
|
|
{2, 0}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(running_ts_sz, &batch);
|
|
|
|
// All `check_mode` pass with OK status and `batch` not checked or updated.
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true));
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
ASSERT_TRUE(new_batch.get() == nullptr);
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest,
|
|
AllInconsistentColumnFamiliesDropped) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{2, 0}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)},
|
|
{3, sizeof(char)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(record_ts_sz, &batch);
|
|
|
|
// All `check_mode` pass with OK status and `batch` not checked or updated.
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true));
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
ASSERT_TRUE(new_batch.get() == nullptr);
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest, InvolvedColumnFamiliesConsistent) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, sizeof(uint64_t)},
|
|
{2, sizeof(char)}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(record_ts_sz, &batch);
|
|
|
|
// All `check_mode` pass with OK status and `batch` not updated.
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true));
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
ASSERT_TRUE(new_batch.get() == nullptr);
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest,
|
|
InconsistentColumnFamilyNeedsTimestampStripping) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, 0}, {2, sizeof(char)}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(record_ts_sz, &batch);
|
|
|
|
// kVerifyConsistency doesn't tolerate inconsistency for running column
|
|
// families.
|
|
ASSERT_TRUE(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true)
|
|
.IsInvalidArgument());
|
|
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
ASSERT_TRUE(new_batch.get() != nullptr);
|
|
CheckContentsWithTimestampStripping(batch, *new_batch, sizeof(uint64_t),
|
|
std::nullopt /* dropped_cf */);
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest,
|
|
InconsistentColumnFamilyNeedsTimestampPadding) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, sizeof(uint64_t)}};
|
|
// Make `record_ts_sz` not contain zero timestamp size entries to follow the
|
|
// behavior of actual WAL log timestamp size record.
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz;
|
|
WriteBatch batch;
|
|
CreateWriteBatch({{1, 0}}, &batch);
|
|
|
|
// kVerifyConsistency doesn't tolerate inconsistency for running column
|
|
// families.
|
|
ASSERT_TRUE(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true)
|
|
.IsInvalidArgument());
|
|
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
ASSERT_TRUE(new_batch.get() != nullptr);
|
|
CheckContentsWithTimestampPadding(batch, *new_batch, sizeof(uint64_t));
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest,
|
|
InconsistencyReconcileCopyOverDroppedColumnFamily) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, 0}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)},
|
|
{2, sizeof(char)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(record_ts_sz, &batch);
|
|
std::unique_ptr<WriteBatch> new_batch(nullptr);
|
|
|
|
// kReconcileInconsistency tolerate inconsistency for dropped column family
|
|
// and all related entries copied over to the new WriteBatch.
|
|
ASSERT_OK(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true, &new_batch));
|
|
|
|
ASSERT_TRUE(new_batch.get() != nullptr);
|
|
CheckContentsWithTimestampStripping(batch, *new_batch, sizeof(uint64_t),
|
|
std::optional<uint32_t>(2));
|
|
}
|
|
|
|
TEST_F(HandleTimestampSizeDifferenceTest, UnrecoverableInconsistency) {
|
|
UnorderedMap<uint32_t, size_t> running_ts_sz = {{1, sizeof(char)}};
|
|
UnorderedMap<uint32_t, size_t> record_ts_sz = {{1, sizeof(uint64_t)}};
|
|
WriteBatch batch;
|
|
CreateWriteBatch(record_ts_sz, &batch);
|
|
|
|
ASSERT_TRUE(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kVerifyConsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true)
|
|
.IsInvalidArgument());
|
|
|
|
ASSERT_TRUE(HandleWriteBatchTimestampSizeDifference(
|
|
&batch, running_ts_sz, record_ts_sz,
|
|
TimestampSizeConsistencyMode::kReconcileInconsistency,
|
|
/* seq_per_batch */ false, /* batch_per_txn */ true)
|
|
.IsInvalidArgument());
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest, EnableUserDefinedTimestamps) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* new_comparator = test::BytewiseComparatorWithU64TsWrapper();
|
|
const Comparator* old_comparator = BytewiseComparator();
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
false /*new_persist_udt*/, true /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_TRUE(mark_sst_files);
|
|
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
false /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_TRUE(mark_sst_files);
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest,
|
|
EnableUserDefinedTimestampsNewPersistUDTFlagIncorrect) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* new_comparator = test::BytewiseComparatorWithU64TsWrapper();
|
|
const Comparator* old_comparator = BytewiseComparator();
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
true /*new_persist_udt*/, true /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
true /*new_persist_udt*/, false /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest, DisableUserDefinedTimestamps) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* new_comparator = ReverseBytewiseComparator();
|
|
const Comparator* old_comparator =
|
|
test::ReverseBytewiseComparatorWithU64TsWrapper();
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
false /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
true /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest,
|
|
DisableUserDefinedTimestampsOldPersistUDTFlagIncorrect) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* new_comparator = BytewiseComparator();
|
|
const Comparator* old_comparator = test::BytewiseComparatorWithU64TsWrapper();
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
false /*new_persist_udt*/, true /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
true /*new_persist_udt*/, true /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest, UserComparatorUnchanged) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* ucmp_without_ts = BytewiseComparator();
|
|
const Comparator* ucmp_with_ts = test::BytewiseComparatorWithU64TsWrapper();
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_without_ts, std::string(ucmp_without_ts->Name()),
|
|
false /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_without_ts, std::string(ucmp_without_ts->Name()),
|
|
true /*new_persist_udt*/, true /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_without_ts, std::string(ucmp_without_ts->Name()),
|
|
true /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_without_ts, std::string(ucmp_without_ts->Name()),
|
|
false /*new_persist_udt*/, true /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_with_ts, std::string(ucmp_with_ts->Name()), true /*new_persist_udt*/,
|
|
true /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
ASSERT_OK(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_with_ts, std::string(ucmp_with_ts->Name()),
|
|
false /*new_persist_udt*/, false /*old_persist_udt*/, &mark_sst_files));
|
|
ASSERT_FALSE(mark_sst_files);
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_with_ts, std::string(ucmp_with_ts->Name()),
|
|
true /*new_persist_udt*/, false /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
ucmp_with_ts, std::string(ucmp_with_ts->Name()),
|
|
false /*new_persist_udt*/, true /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
}
|
|
|
|
TEST(ValidateUserDefinedTimestampsOptionsTest, InvalidUserComparatorChange) {
|
|
bool mark_sst_files = false;
|
|
const Comparator* new_comparator = BytewiseComparator();
|
|
const Comparator* old_comparator = ReverseBytewiseComparator();
|
|
ASSERT_TRUE(ValidateUserDefinedTimestampsOptions(
|
|
new_comparator, std::string(old_comparator->Name()),
|
|
false /*new_persist_udt*/, true /*old_persist_udt*/,
|
|
&mark_sst_files)
|
|
.IsInvalidArgument());
|
|
}
|
|
|
|
TEST(GetFullHistoryTsLowFromU64CutoffTsTest, Success) {
|
|
std::string cutoff_ts;
|
|
uint64_t cutoff_ts_int = 3;
|
|
PutFixed64(&cutoff_ts, 3);
|
|
Slice cutoff_ts_slice = cutoff_ts;
|
|
std::string actual_full_history_ts_low;
|
|
GetFullHistoryTsLowFromU64CutoffTs(&cutoff_ts_slice,
|
|
&actual_full_history_ts_low);
|
|
|
|
std::string expected_ts_low;
|
|
PutFixed64(&expected_ts_low, cutoff_ts_int + 1);
|
|
ASSERT_EQ(expected_ts_low, actual_full_history_ts_low);
|
|
}
|
|
} // namespace ROCKSDB_NAMESPACE
|
|
|
|
int main(int argc, char** argv) {
|
|
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
|
|
::testing::InitGoogleTest(&argc, argv);
|
|
return RUN_ALL_TESTS();
|
|
}
|