A unified file format for WAL, Blob, and related data in RocksDB
Status: Implementation in progress | Format name: "blog" (portmanteau of "blob" + "log")
| Term | Scope | Definition |
|---|---|---|
| Record | Blog file | A top-level unit in the blog file body: blob record, WriteBatch record, preamble-start record, footer record. |
| Operation (op) | WriteBatch | An entry within a WriteBatch: Put op, Delete op, Merge op, etc. |
| Compact format | Record | A record with varint length ≤ 3 bytes. Omits the type byte (type from header). |
| Full format | Record | A record with varint length > 3 bytes. Includes explicit type, pre-payload compression type, and prefix checksum. |
A blog file consists of three regions: a fixed header with a property section, a body of records, and an optional footer composed of footer records.
| Offset | Size | Field |
|---|---|---|
| 0 | 12 | File magic: kBlogFileMagic -- opaque 12 bytes ("Blog" ASCII + XXH3_64bits("Blog")) |
| 12 | 1 | Schema version (currently 0). Newer versions are rejected by older readers. |
| 13 | 1 | Checksum type (ChecksumType enum: 0=none, 1=CRC32c, 4=XXH3). Configured via DBOptions::blog_checksum. |
| 14 | 1 | Compact record type (BlogRecordType -- type for all compact-format records) |
| 15 | 1 | Flags: bit 0 = kBlogFileRecycled (file may contain trailing stale data) |
| 16 | 10 | Escape sequence: bytes [0-5] random (byte[0] not 0x00/0xFF), bytes [6-9] = lower32(XXH3_64bits_withSeed(bytes[0-5], kBlogEscapeSeqSeed)). First 4 bytes also serve as incarnation ID (little-endian via DecodeFixed32). |
| 26 | 6 | Reserved (zero) |
| 32 | 4 | Property section size (uint32 LE) -- includes 5-byte trailer |
| 36 | 4 | Fixed header checksum -- covers bytes [0, 36) |
Immediately follows the fixed header. Contains named key-value pairs encoded as:
[name_len: varint32] [name: name_len bytes] [value_len: varint32] [value: value_len bytes] ...
Followed by a 5-byte trailer: [compression_type: 1B][checksum: 4B] (same as block-based table). The checksum covers the property data + compression_type, with a context checksum modifier using the incarnation ID at offset kBlogFileFixedHeaderSize.
After the property section, arbitrarily long padding (0x00 or 0xFF, differing from the last meaningful byte) aligns the first record's escape sequence. This supports stronger alignment requirements (flush/sync boundaries) or recycling files of all zeros.
| First letter | Meaning |
|---|---|
Uppercase (e.g. "CompressionCompatibilityName") | Required -- reader must recognize or reject the file |
Lowercase (e.g. "role") | Ignorable -- reader may skip unrecognized properties |
| Name | Required? | Description |
|---|---|---|
"CompressionCompatibilityName" | Required (if any compression) | Compression manager compatibility name (e.g. "builtin_v2") |
"WriteBatchStreamingCompressionType" | Required (if streaming active) | 2-digit hex of CompressionType for streaming WriteBatch compression (e.g. "07" for ZSTD). Presence means WriteBatch records with kStreamingCompressionSentinel (0x7F) trailer are part of a cross-record streaming context. |
"TimestampSize" | Required (if >0) | Size of user timestamps |
"PredecessorWalInfo" | Required (if tracking) | Predecessor WAL metadata for verification |
"role" | Ignorable | File purpose hint: "wal", "blob", "rlog" |
"compressionSettings" | Ignorable | Human-readable compression settings for debugging |
"creationTime" | Ignorable | Unix timestamp of file creation |
"creator" | Ignorable | Host/process identifier for debugging |
Escape sequences must be 32-bit aligned in the file. The first 4 bytes (little-endian) serve as the incarnation ID for context checksums.
kBlogFileRecycled flag in the header indicates recycling is expected.kSkipAnyCorruptedRecords).| Value | Name | Usage |
|---|---|---|
| Body record types (0x01-0x7F) | ||
| 0x01 | kBlogBlobRecord | Blob data (typically a separated value) |
| 0x02 | kBlogPreambleStartRecord | Preamble marker (stub) |
| 0x03 | kBlogWriteBatchRecord | Serialized WriteBatch |
| 0x04-0x7F | Reserved for future body record types | |
| Footer record types (0xF0-0xFF, in expected order of occurrence with gaps) | ||
| 0xF0 | kBlogFooterIndexRecord | Large footer metadata (sparse index) |
| 0xF1-0xF3 | Reserved for future footer data records | |
| 0xF4-0xF7 | Reserved for future footer data records | |
| 0xF8 | kBlogFooterPropertiesRecord | Small footer metadata (named properties) |
| 0xF9-0xFB | Reserved | |
| 0xFC | kBlogFooterFileChecksumInfo | Full file checksum state dump. Allows inferring the file checksum from just reading the tail. Placeholder for future use. |
| 0xFD-0xFE | Reserved | |
| 0xFF | kBlogFooterLocatorRecord | Footer locator with relative offsets. Must be the last record in the file. |
length = 0: No payload and no 5-byte trailer. Record is escape_seq + prefix fields + padding.length > 0: Payload of length bytes followed by 5-byte trailer.length = 2^63 - 1: Unspecified size. Reader scans forward for next escape sequence.No type byte -- type comes from the header's compact_record_type field. Used for payloads < 16 KiB.
The prefix_checksum covers (varint + type + compression_type), with a context checksum modifier for defense-in-depth. Validates the length before attempting a potentially large read. The pre-payload compression_type enables streaming decompression without seeking to the trailer.
Records that don't match the compact_record_type (e.g. footer records, preamble-start) always use full format by encoding the varint with ≥ 3 bytes (irregular encoding).
All checksums in blog files (fixed header, property section trailer, record prefix, record trailer) use context checksum modifiers incorporating the incarnation ID, providing defense-in-depth for misplaced data detection even with kNoChecksum.
Each record's trailer compression_type byte indicates the compression algorithm used for that record's payload. The actual type comes from Compressor::CompressBlock output, not the configured type. This is the same approach used by SST block-based table builders.
When the header property WriteBatchStreamingCompressionType is set, WriteBatch record payloads may use streaming compression across records. Such records use kStreamingCompressionSentinel (0x7F) as their trailer's compression_type byte, indicating "compressed using the streaming algorithm declared in the header."
A record with kNoCompression (0x00) in the trailer remains individually uncompressed even when streaming is active, enabling dynamic on/off switching based on compression effectiveness.
Non-WriteBatch records (blob records, footer records) are always outside the streaming context and use per-record compression, keeping blob data randomly readable.
WriteBatchStreamingCompressionType property is required (uppercase first letter), so older readers that don't understand it will reject the file rather than misinterpret streaming-compressed payloads.
Padding fills the gap between a record's trailer (or the header property section) and the next escape sequence. The padding byte value is chosen to differ from the last meaningful byte, enabling unambiguous scanning.
| Last meaningful byte | Pad with | Mandatory? |
|---|---|---|
0x00 | 0xFF | Yes -- at least 1 byte required |
0xFF | 0x00 | Yes -- at least 1 byte required |
| Anything else | 0x00 (writer's choice) | Only if needed for alignment |
Padding may be arbitrarily long for stronger alignment requirements (e.g. flush/sync boundaries) or for recycling a file of all zeros. The reader skips padding by scanning forward over 0x00/0xFF bytes in 4-byte-aligned chunks until finding a byte that is neither, which is the first byte of the next escape sequence (guaranteed to be neither 0x00 nor 0xFF).
ChecksumModifierForContext(incarnation_id, record_offset) -- the same mechanism as block-based SST files.
Context checksums provide defense-in-depth for recycling detection and general data integrity verification with zero per-record overhead.
The checksum type is configured from the CF's BlockBasedTableOptions::checksum setting (for blob files) or the default CF's setting (for WAL files). This is a reasonable but hopefully temporary way of configuring blog file internal checksums; a dedicated option may be added later.
The footer is composed of regular records using the full record format, maintaining simple forward parsability. A file is cleanly sealed when a valid footer locator record is the last record. Footer records appear in the order listed below.
[record_type: 1B] [relative_offset_in_4B_units: uint32 LE] ...
First entry: distance from locator's escape_seq backward to the preceding footer record. Subsequent: from the previous entry backward to the next.
| Name | Description |
|---|---|
"blobCount" | Number of blob records |
"totalBlobBytes" | Total uncompressed blob payload bytes |
"totalBlobBytesCompressed" | Total compressed blob payload bytes on disk |
"totalWriteBatchBytes" | Total uncompressed WriteBatch payload bytes |
"totalWriteBatchBytesCompressed" | Total compressed WriteBatch payload bytes on disk |
"minSequence" | Minimum sequence number |
"maxSequence" | Maximum sequence number |
The escape sequence is the primary mechanism for detecting WAL recycling and for non-linear data recovery. The kBlogFileRecycled flag in the header indicates the file may contain trailing stale data from a previous incarnation.
| Mode | Behavior on Undecodable Data | Escape Sequence Role |
|---|---|---|
kAbsoluteConsistency |
Error on any failure, including incomplete trailing records. If recycled and no escape sequence found in trailing data, accept as clean end. | Distinguishes stale recycled data from corruption. |
kTolerateCorruptedTailRecords |
Incomplete trailing record (Incomplete status): tolerate, stop at last complete record. Corruption: scan for escape sequence. Not found = clean end (recycled or preallocated). Found with valid record beyond = mid-file corruption (error). | Distinguishes tail damage from mid-file corruption. |
kPointInTimeRecovery (default) |
Stop at first inconsistency. Incomplete trailing records tolerated. Recycled trailing data accepted. | Diagnostic improvement. |
kSkipAnyCorruptedRecords |
On corruption, scan forward for next escape sequence. If found, try to decode and verify; resume if valid. If not found, stop. | Essential for record discovery after corruption. |
A blob preamble is a sequence of blob records preceded by a preamble-start record (type 0x02, full format, length=0) and followed by the associated WriteBatch record. The blob records contain separated values referenced by file_number = 0 blob references in the WriteBatch.
This functionality is stubbed in the current implementation: the preamble-start record type is defined and can be written/read, but the full preamble flow (in-memory serialization, blob reference encoding) is deferred.
Deferred. The blog format supports a "rlog" role hint for replication log storage, with sparse indexes in footer records and in-memory index maintenance. See design notes in the original proposal.
| Question | Resolution |
|---|---|
| Escape sequence size | 10 bytes: 6 random + 4 derived. First byte not 0x00/0xFF. |
| Header/footer encoding | 40-byte fixed header + named property section with 5-byte trailer. Footer uses regular full-format records: index (0xF0), properties (0xF8), file checksum info (0xFC, placeholder), locator (0xFF). Numbered in expected occurrence order with gaps for future types. |
| Preamble-start record format | Full format, length=0, type 0x02. Minimal -- just the type tag. |
| WriteBatch record framing | Same compact/full format as blob records, type 0x03. |
| Compact vs full format | Compact: varint ≤ 2B (payload < 16 KiB), matches compact_record_type, no type byte. Full: varint > 2B (possibly irregular), includes type + pre-payload compression_type + prefix_checksum with context modifier. |
| Checksum context | All checksums (fixed header, property section, record prefix, record trailer) use context checksum modifiers with the incarnation ID, ensuring nonzero stored values even with kNoChecksum. |
| Schema version | Starts at 0. Newer versions are rejected by older readers (returns NotSupported). Checksum is verified before schema version is checked. |
| Streaming compression sentinel | kStreamingCompressionSentinel = 0x7F defined in CompressionType enum (compression_type.h). Used in blog record trailers for streaming-compressed WriteBatch payloads. |
use_blog_format_for_wals (bool, default false): New WAL files use blog format.use_blog_format_for_blobs (bool, default false): New blob files use blog format.blog_checksum (ChecksumType, default kXXH3): Checksum algorithm for blog format files. ChecksumType enum moved to include/rocksdb/checksum_type.h (from table.h) for use by both table and DB options.All are immutable DBOptions. Existing legacy WAL and blob files are always readable regardless of these settings.
MutableCFOptions::compression_manager). Legacy blob files continue to use the builtin manager for backward compatibility.Compressor::CompressBlock. The actual CompressionType from CompressBlock output is stored in the record trailer.WriteBatchStreamingCompressionType property, kStreamingCompressionSentinel 0x7F trailer value) but is not yet enabled by default.