docs/source/administration/object-management/object-versioning.rst

Bucket Versioning

minio

Table of Contents

• Versioning overview
• Versioning lab

Overview

MinIO supports keeping multiple "versions" of an object in a single bucket.

When enabled, versioning allows MinIO to keep multiple iterations of the same object. Write operations which would normally overwrite an existing object instead result in the creation of a new versioned object. MinIO versioning protects from unintended overwrites and deletions while providing support for "undoing" a write operation. Bucket versioning is a prerequisite for configuring object locking and retention rules <minio-object-locking>.

For versioned buckets, a write operation that mutates an object results in a new version of that object with a unique version ID. MinIO marks the "latest" version of the object that clients retrieve by default. Clients can then explicitly choose to list, retrieve, or remove a specific object version.

Define object expiration <minio-lifecycle-management-create-expiry-rule> rules to remove versions of objects no longer needed, such as by the number of versions or the date of versions.

Read Operations on Versioned Objects

Review each of the four images in this series to see how MinIO retrieves objects in a versioned bucket. Use the arrows on either side of the images to navigate from one to the next.

1

Object with Single Version

MinIO adds a unique version ID to each object as part of write operations.

Object with Multiple Versions

MinIO retains all versions of an object and marks the most recent version as the "latest".

Retrieving the Latest Object Version

A read operation request without a version ID returns the latest version of the object.

Retrieving a Specific Object Version

Include the version ID to retrieve a specific version of an object during a read operation.

MinIO Server RELEASE.2023-05-04T21-44-30Z

MinIO does not create versions for creation, mutation, or deletion of explicit directory objects ("prefixes"). Objects created within that explicit directory object retain normal versioning behavior.

MinIO implicitly determines prefixes from object paths. Explicit prefix creation typically only occurs with Spark and similar workloads which apply legacy POSIX/HDFS directory creation behavior within the S3 context.

Versioning is Per-Namespace

MinIO uses the full namespace (the bucket and path to an object) for each object as part of determining object uniqueness. For example, all of the following namespaces are "unique" objects, where mutations of each object result in the creation of new object versions at that namespace:

databucket/object.blob
databucket/blobs/object.blob
blobbucket/object.blob
blobbucket/blobs/object.blob

While object.blob might be the same binary across all namespaces, MinIO only enforces versioning with a specific namespace and therefore considers each object.blob above as distinct and unique.

Versioning and Storage Capacity

MinIO does not perform incremental or differential-type versioning. For mutation-heavy workloads, this may result in substantial drive usage by older or aged object versions.

For example, consider a 1GB object containing log data. An application appends 100MB of data to the log and uploads to MinIO. MinIO would then contain both the 1GB and 1.1GB versions of the object. If the application repeated this process every day for 10 days, the bucket would eventually contain more than 14GB of data associated to a single object.

MinIO supports configuring configuring object lifecycle management rules <minio-lifecycle-management> to automatically expire or transition aged object versions and free up storage capacity. For example, you can configure a rule to automatically expire object versions 90 days after they become non-current (i.e. no longer the "latest" version of that object). See MinIO Object Expiration <minio-lifecycle-management-expiration> for more information.

You can alternatively perform manual removal of object versions using the following commands:

• mc rm --versions - Removes all versions of an object.
• mc rm --versions --older-than <mc rm --older-than> - Removes all versions of an object older than the specified calendar date.

RELEASE.2024-04-18T19-09-19Z

MinIO emits a warning if the cumulative size of versions for any single object exceeds 1TiB.

Version ID Generation

MinIO generates a unique and immutable identifier for each versioned object as part of write operations. Each object version ID consists of a 128-bit fixed-size UUIDv4 <4122#section-4.4>. UUID generation is sufficiently random to ensure high likelihood of uniqueness for any environment, are computationally difficult to guess, and do not require centralized registration process and authority to guarantee uniqueness.

MinIO does not support client-managed version ID allocation. All version ID generation is handled by the MinIO server process.

For objects created while versioning is disabled or suspended, MinIO uses a null version ID. You can access or remove these objects by specifying null as the version ID as part of S3 operations.

Versioned Delete Operations

Performing a DELETE operation on a versioned object creates a 0-byte DeleteMarker as the latest version of that object. For objects where the latest version is a DeleteMarker, clients must specify versioning flags or identifiers to perform GET/HEAD/LIST/DELETE operations on a prior version of that object. The default server behavior omits DeleteMarker objects from consideration for unversioned operations.

MinIO can utilize Lifecycle Management expiration rules <minio-lifecycle-management-expiration> to automatically remove versioned objects permanently. Otherwise, use manual DELETE operations to permanently remove non-current versioned objects or DeleteMarker objects.

MinIO Implements Idempotent Delete Markers

RELEASE.2022-08-22T23-53-06Z

Standard S3 implementations can create multiple sequential delete markers for the same object when processing simple DeleteObject requests with no version identifier. See the S3 docs for details on managing delete markers <ManagingDelMarkers.html#RemDelMarker>.

MinIO diverges from standard S3 implementation by avoiding this potential duplication of delete markers. When processing a Delete request with no version identifier, MinIO creates at most one Delete Marker for the specified object. MinIO does not share S3's behavior in creating multiple sequential delete markers.

To permanently delete an object version, perform the DELETE operation and specify the version ID of the object to delete. Versioned delete operations are irreversible.

1

Deleting an Object

Performing a DELETE operation on a versioned object produces a DeleteMarker for that object.

Reading a Deleted Object

Clients by default retrieve the "latest" object version. MinIO returns a 404-like response if the latest version is a DeleteMarker.

Retrieve Previous Version of Deleted Object

Clients can retrieve any previous version of the object by specifying the version ID, even if the "Latest" version is a DeleteMarker.

Delete a Specific Object Version

Clients can delete a specific object version by specifying the version ID as part of the DELETE operation. Deleting a specific version is permanent and does not result in the creation of a DeleteMarker.

The following mc commands operate on DeleteMarkers or versioned objects:

• Use mc ls --versions to view all versions of an object, including delete markers.
• Use mc cp --version-id=UUID ... <mc cp --version-id> to retrieve the version of the "deleted" object with matching UUID.
• Use mc rm --version-id=UUID ... <mc rm --version-id> to delete the version of the object with matching UUID.
• Use mc rm --versions to delete all versions of an object.

Tutorials

Enable Bucket Versioning

You can enable versioning using the MinIO Console, the MinIO mc CLI, or using an S3-compatible SDK.

MinIO Console

Select the Buckets section of the MinIO Console to access bucket creation and management functions. You can use the search Search bar to filter the list.

Each bucket row has a Manage button that opens the management view for that bucket.

Toggle the Versioning field to enable versioning on the bucket.

The MinIO Console also supports enabling versioning as part of bucket creation. See minio-console-buckets for more information on bucket management using the MinIO Console.

MinIO CLI

Use the mc version enable command to enable versioning on an existing bucket:

mc version ALIAS/BUCKET

• Replace ALIAS with the alias <mc alias> of a configured MinIO deployment.
• Replace BUCKET with the target bucket <mc version enable ALIAS> on which to enable versioning.

Objects created prior to enabling versioning have a null version ID <minio-bucket-versioning-id>.

Exclude a Prefix From Versioning

You can exclude certain prefixes <minio-admin-concepts-organize-objects> from versioning using the MinIO Client <minio-client>. This is useful for Spark/Hadoop workloads or others that initially create objects with temporary prefixes.

Replication and Object Locking Require Versioning

MinIO requires versioning to support replication. Objects in excluded prefixes do not replicate to any peer site or remote site.

MinIO does not support excluding prefixes from versioning on buckets with object locking enabled <minio-object-locking>.

• Use mc version enable with the ~mc version enable --excluded-prefixes option:

  mc version enable --excluded-prefixes "prefix1, prefix2" ALIAS/BUCKET

  • Replace ALIAS with the alias <mc alias> of a configured MinIO deployment.
  • Replace BUCKET with the name of the bucket <UsingBucket.html> you want to exclude prefixes <minio-admin-concepts-organize-objects> for.

The list of ~mc version enable --excluded-prefixes prefixes match all objects containing the specified strings in their prefix or name, similar to a regular expression of the form prefix*. To match objects by prefix only, use prefix/*.

For example, the following command excludes any objects containing _test or _temp in their prefix or name from versioning:

mc version enable --excluded-prefixes "_test, _temp" local/my-bucket

You can exclude up to 10 prefixes for each bucket. To add or remove prefixes, repeat the mc version enable command with an updated list. The new list of prefixes replaces the previous one.

To view the currently excluded prefixes, use mc version info with the --json option:

mc version info ALIAS/BUCKET --json

The command output resembles the following, with the list of excluded prefixes in the ExcludedPrefixes property:

$ mc version info local/my-bucket --json
{
 "Op": "info",
 "status": "success",
 "url": "local/my-bucket",
 "versioning": {
  "status": "Enabled",
  "MFADelete": "",
  "ExcludedPrefixes": [
   "prefix1, prefix2"
  ]
 }
}

To disable prefix exclusion and resume versioning all prefixes, repeat the mc version enable command without ~mc version enable --excluded-prefixes:

mc version enable ALIAS/BUCKET

Exclude Folders from Versioning

You can exclude folders from versioning using the MinIO Client <minio-client>.

Replication and Object Locking Require Versioning

MinIO requires versioning to support replication. Objects in excluded folders do not replicate to any peer site or remote site.

MinIO does not support excluding folders from versioning on buckets with object locking enabled <minio-object-locking>.

Object locking

Buckets with object locking enabled <minio-object-locking> require versioning and do not support excluding folders.

• Use mc version enable with the ~mc version enable --exclude-folders option to exclude objects with names ending in / from versioning:

  mc version enable --exclude-folders ALIAS/BUCKET

  • Replace ALIAS with the alias <mc alias> of a configured MinIO deployment.
  • Replace BUCKET with the bucket <UsingBucket.html> you want to exclude folders <minio-admin-concepts-organize-objects> for.

To check whether folders are versioned for a bucket, use the mc version enable command with the --json option. If the ExcludeFolders property is true, folders in that bucket are not versioned.

mc version enable --excluded-prefixes ALIAS/BUCKET --json

The command output resembles the following:

$ mc version info local/my-bucket --json
{
 "Op": "info",
 "status": "success",
 "url": "local/my-bucket",
 "versioning": {
  "status": "Enabled",
  "MFADelete": "",
  "ExcludeFolders": true
 }
}

To disable folder exclusion and resume versioning all folders, repeat the mc version enable command without ~mc version enable --exclude-folders:

mc version enable ALIAS/BUCKET

Suspend Bucket Versioning

You can suspend bucket versioning at any time using the MinIO Console, the MinIO mc CLI, or using an S3-compatible SDK.

MinIO Console

Select the Buckets section of the MinIO Console to access bucket creation and management functions. You can use the search Search bar to filter the list.

Each bucket row has a Manage button that opens the management view for that bucket.

Select the Versioning field and follow the instructions to suspend versioning in the bucket.

See minio-console-buckets for more information on bucket management using the MinIO Console.

MinIO CLI

Use the mc version suspend command to enable versioning on an existing bucket:

mc version suspend ALIAS/BUCKET

• Replace ALIAS with the alias <mc alias> of a configured MinIO deployment.
• Replace BUCKET with the target bucket <mc mb ALIAS> on which to disable versioning.

Objects created while versioning is suspended are assigned a null version ID <minio-bucket-versioning-id>. Any mutations to an object while versioning is suspended result in overwriting that null versioned object. MinIO does not remove or otherwise alter existing versioned objects as part of suspending versioning. Clients can continue interacting with any existing object versions in the bucket.