mirror of
https://github.com/facebook/rocksdb.git
synced 2026-10-23 08:09:02 +03:00
Summary: BlockBuilder was somewhat inconsistent in its treatment of Slices whose size exceeds 4GB, which in a random corruption case could lead to (for example) serializing only the bottom 32 bits of a value size (uncorrupted) but appending the full multi-GB (corrupted) size. Because this is inner loop code, we don't want to pay CPU for extra conversions, data movements, or Status plumbing (already ruled out by Xingbo). In this change we make BlockBuilder more internally consistent and lift the 32-bit size requirement to callers (of a few functions specifically, for now). To ensure that's satisfied, I've added additional checks near the perimeter of RocksDB to ensure keys and values do not exceed 4GB, plus an extra random corruption or backstop check in BlockBasedTableBuilder. In detail, BlockBuilder (block_builder.cc/h): * Add API comments documenting the < 4GB assumption on Add/AddWithLastKey * Add debug assertions verifying input slice sizes < 4GB * Simplify AddWithLastKeyImpl to use uint32_t locals, reducing static_cast * Use only bottom 32 bits when appending derivative slices for consistency * Update MaybeStripTimestampFromKey to also truncate to 32-bit size * Add FIXME comments where buffer_/values_buffer_ sizes are truncated BlockBasedTableBuilder (block_based_table_builder.cc): * Add value size check (> uint32_t max) as a safety net against random corruption, since we have seen such corruptions in production WriteBatch (write_batch.cc): * Tighten existing key size checks to account for kNumInternalBytes (8-byte internal key suffix), using new kMaxWriteBatchKeySize constant * Add missing size checks to Delete, SingleDelete, and DeleteRange (both Slice and SliceParts variants) SstFileWriter (sst_file_writer.cc): * Add key and value size checks in AddImpl and DeleteRangeImpl, which bypass WriteBatch and go directly to the table builder MergeHelper (merge_helper.cc): * Add merge result size check (> uint32_t max) in both overloads of TimedFullMergeImpl, returning Corruption if exceeded * Add PartialMergeMulti result size check in MergeUntil CompactionIterator (compaction_iterator.cc): * Add size check on compaction filter output values (kChangeValue and kChangeWideColumnEntity), returning Corruption if > 4GB meta_blocks.cc: * Use Slice with uint32_t-truncated sizes in PropertyBlockBuilder::Finish to handle potentially oversized user property collector output Needed follow-up: * Check for blocks that are mis-encoded due to overflowing 32 bits for restart point offsets (and similar). See FIXME comments in block_builder.cc for why this is tricky. Pull Request resolved: https://github.com/facebook/rocksdb/pull/14461 Test Plan: New extreme-size unit tests, along with some testing infrastructure improvements: Added test::HasBigMem() (in test_util/testharness.h) which returns true when the system has ≥128GB RAM (via sysconf(_SC_PHYS_PAGES) on Linux/macOS) or when ROCKSDB_BIGMEM_TESTS is set. All extreme-size tests use HasBigMem() to skip gracefully on smaller machines rather than being permanently disabled. Where possible, tests use MemMapping::AllocateLazyZeroed() to supply large keys/values as Slices backed by anonymous mmap (cleaner with new MemMapping::AsSlice()). On Linux, read-only access to these pages maps to the shared kernel zero page, so the source data consumes no physical RAM — only the destination copy (e.g., WriteBatch::rep_) materializes, cutting peak memory roughly in half vs. std::string. Tests (all enabled, skip via HasBigMem()): ./write_batch_test --gtest_filter='*LargeKeyValueSizeLimit*' ./external_sst_file_basic_test --gtest_filter='*LargeSizeSstFileWriter*' ./merge_test --gtest_filter='*LargeMergeResultRejected*' ./merge_helper_test --gtest_filter='*LargePartialMergeResultRejected*' ./db_compaction_test --gtest_filter='*CompactionFilterLargeValueRejected*' All 5 pass (verified on a 128GB+ machine). On smaller machines, all 5 bypass cleanly with "insufficient memory for reliable continuous testing". Reviewed By: xingbowang Differential Revision: D96521899 Pulled By: pdillinger fbshipit-source-id: 70f4b5e6a23ab074d60e653fbb7ddc5edbe162ab
131 lines
4.8 KiB
C++
131 lines
4.8 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#pragma once
|
|
|
|
#ifdef OS_AIX
|
|
#include "gtest/gtest.h"
|
|
#else
|
|
#include <gtest/gtest.h>
|
|
#endif
|
|
|
|
// A "skipped" test has a specific meaning in Facebook infrastructure: the
|
|
// test is in good shape and should be run, but something about the
|
|
// compilation or execution environment means the test cannot be run.
|
|
// Specifically, there is a hole in intended testing if any
|
|
// parameterization of a test (e.g. Foo/FooTest.Bar/42) is skipped for all
|
|
// tested build configurations/platforms/etc.
|
|
//
|
|
// If GTEST_SKIP is available, use it. Otherwise, define skip as success.
|
|
//
|
|
// The GTEST macros do not seem to print the message, even with -verbose,
|
|
// so these print to stderr. Note that these do not exit the test themselves;
|
|
// calling code should 'return' or similar from the test.
|
|
#ifdef GTEST_SKIP_
|
|
#define ROCKSDB_GTEST_SKIP(m) \
|
|
do { \
|
|
fputs("SKIPPED: " m "\n", stderr); \
|
|
GTEST_SKIP_(m); \
|
|
} while (false) /* user ; */
|
|
#else
|
|
#define ROCKSDB_GTEST_SKIP(m) \
|
|
do { \
|
|
fputs("SKIPPED: " m "\n", stderr); \
|
|
GTEST_SUCCESS_("SKIPPED: " m); \
|
|
} while (false) /* user ; */
|
|
#endif
|
|
|
|
// We add "bypass" as an alternative to ROCKSDB_GTEST_SKIP that is allowed to
|
|
// be a permanent condition, e.g. for intentionally omitting or disabling some
|
|
// parameterizations for some tests. (Use _DISABLED at the end of the test
|
|
// name to disable an entire test.)
|
|
#define ROCKSDB_GTEST_BYPASS(m) \
|
|
do { \
|
|
fputs("BYPASSED: " m "\n", stderr); \
|
|
GTEST_SUCCESS_("BYPASSED: " m); \
|
|
} while (false) /* user ; */
|
|
|
|
// Avoid "loss of precision" warnings when passing in 64-bit integers
|
|
#define EXPECT_NEAR2(val1, val2, abs_error) \
|
|
EXPECT_NEAR(static_cast<double>(val1), static_cast<double>(val2), \
|
|
static_cast<double>(abs_error))
|
|
|
|
#include <string>
|
|
|
|
#include "port/stack_trace.h"
|
|
#include "rocksdb/env.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
namespace test {
|
|
|
|
// Return the directory to use for temporary storage.
|
|
std::string TmpDir(Env* env = Env::Default());
|
|
|
|
// A path unique within the thread
|
|
std::string PerThreadDBPath(std::string name);
|
|
std::string PerThreadDBPath(Env* env, std::string name);
|
|
std::string PerThreadDBPath(std::string dir, std::string name);
|
|
|
|
// Return a randomization seed for this run. Typically returns the
|
|
// same number on repeated invocations of this binary, but automated
|
|
// runs may be able to vary the seed.
|
|
int RandomSeed();
|
|
|
|
// Returns true if the system has enough memory to run expensive
|
|
// large-allocation tests (~128GB+ RAM or environment variable
|
|
// ROCKSDB_BIGMEM_TESTS is set). There should generally be only one "big mem"
|
|
// test per unit test file, to reduce OOM risks in parallel runs.
|
|
bool HasBigMem();
|
|
|
|
::testing::AssertionResult AssertStatus(const char* s_expr, const Status& s);
|
|
|
|
#define ASSERT_OK(s) \
|
|
ASSERT_PRED_FORMAT1(ROCKSDB_NAMESPACE::test::AssertStatus, s)
|
|
#define ASSERT_NOK(s) ASSERT_FALSE((s).ok())
|
|
#define EXPECT_OK(s) \
|
|
EXPECT_PRED_FORMAT1(ROCKSDB_NAMESPACE::test::AssertStatus, s)
|
|
#define EXPECT_NOK(s) EXPECT_FALSE((s).ok())
|
|
|
|
// Useful for testing
|
|
// * No need to deal with Status like in Regex public API
|
|
// * No triggering lint reports on use of std::regex in tests
|
|
// * Available in LITE (unlike public API)
|
|
class TestRegex {
|
|
public:
|
|
// These throw on bad pattern
|
|
/*implicit*/ TestRegex(const std::string& pattern);
|
|
/*implicit*/ TestRegex(const char* pattern);
|
|
|
|
// Checks that the whole of str is matched by this regex
|
|
bool Matches(const std::string& str) const;
|
|
|
|
const std::string& GetPattern() const;
|
|
|
|
private:
|
|
class Impl;
|
|
std::shared_ptr<Impl> impl_; // shared_ptr for simple implementation
|
|
std::string pattern_;
|
|
};
|
|
|
|
::testing::AssertionResult AssertMatchesRegex(const char* str_expr,
|
|
const char* pattern_expr,
|
|
const std::string& str,
|
|
const TestRegex& pattern);
|
|
|
|
#define ASSERT_MATCHES_REGEX(str, pattern) \
|
|
ASSERT_PRED_FORMAT2(ROCKSDB_NAMESPACE::test::AssertMatchesRegex, str, pattern)
|
|
#define EXPECT_MATCHES_REGEX(str, pattern) \
|
|
EXPECT_PRED_FORMAT2(ROCKSDB_NAMESPACE::test::AssertMatchesRegex, str, pattern)
|
|
|
|
} // namespace test
|
|
|
|
using test::TestRegex;
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|