diff --git a/requirements.txt b/requirements.txt index 23b96140..9a32e513 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,5 +4,5 @@ sphinx-design == 0.0.12 sphinx-markdown-tables == 0.0.15 Sphinx-Substitution-Extensions == 2020.9.30.0 sphinx-togglebutton === 0.2.3 -sphinxcontrib-images === 0.9.3 +sphinxcontrib-images === 0.9.4 myst-parser === 0.15.1 \ No newline at end of file diff --git a/source/concepts/bucket-versioning.rst b/source/concepts/bucket-versioning.rst index aeb2992f..51344109 100644 --- a/source/concepts/bucket-versioning.rst +++ b/source/concepts/bucket-versioning.rst @@ -10,78 +10,310 @@ Bucket Versioning :local: :depth: 2 +Overview +-------- + MinIO supports keeping multiple "versions" of an object in a single bucket. Write operations which would normally overwrite an existing object instead result in the creation of a new versioned object. MinIO versioning protects from unintended overwrites and deletions while providing support for "undoing" a -write operation. Bucket versioning also supports retention and archive policies. +write operation. Bucket versioning is a prerequisite for configuring +:ref:`object locking and retention rules `. - +For versioned buckets, any write operation that mutates an object results in a +new version of that object with a unique version ID. MinIO marks the "latest" +version of the object that clients retrieve by default. Clients can then +explicitly choose to list, retrieve, or remove a specific object version. -MinIO generates a unique immutable ID for each object. If a ``PUT`` request -contains an object name which duplicates an existing object, MinIO does *not* -overwrite the "older" object. Instead, MinIO retains all object versions while -considering the most recently written "version" of the object as "latest". -Applications retrieve the latest object version by default, but *may* retrieve -any other version in the history of that object. To view all versions of an -object or objects in a bucket, use the :mc-cmd-option:`mc ls versions` command. +.. card-carousel:: 1 -By default, deleting an object does *not* remove all existing versions of -that object. Instead, MinIO places a "delete" marker for the object, such that -applications requesting only the latest object versions do not see the object. -Applications *may* retrieve earlier versions of that object. To completely -delete an object and its entire version history, use the -:mc-cmd-option:`mc rm versions` command. + .. card:: Object with Single Version -Enable Bucket Versioning ------------------------- + .. image:: /images/retention/minio-versioning-single-version.svg + :alt: Object with single version + :align: center -Enabling bucket versioning on a MinIO deployment requires that the deployment -have *at least* four disks. Specifically, Bucket Versioning depends on -:ref:`Erasure Coding `. For MinIO deployments that -meet the disk requirements, use the :mc-cmd:`mc version enable` command to -enable versioning on a specific bucket. + MinIO adds a unique version ID to each object as part of write operations. -The :mc-cmd:`mc version` command *may* work on other S3-compatible services -depending on their implementation of and support for the AWS S3 API. + .. card:: Object with Multiple Versions -Buckets with Existing Content -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + .. image:: /images/retention/minio-versioning-multiple-versions.svg + :alt: Object with Multiple Versions + :align: center -After enabling bucket versioning, MinIO begins generating version IDs for -*new* objects created in the bucket. MinIO does *not* iterate through existing -objects to generate version IDs. Existing unversioned objects in a versioned -bucket have a ``null`` version ID. + MinIO retains all versions of an object and marks the most recent + version as the "latest". -Suspend Bucket Versioning -------------------------- + .. card:: Retrieving the Latest Object Version -You can suspend bucket versioning at any time using -:mc-cmd:`mc version disable`. MinIO retains all existing object versions. During -suspension, MinIO allows overwrites of any *unversioned* object. Applications -can continue referencing any existing object version. + .. image:: /images/retention/minio-versioning-retrieve-latest-version.svg + :alt: Object with Multiple Versions + :align: center -You can later re-enable object versioning on the bucket. MinIO resumes -generating version IDs for *new* objects, and does not retroactively generate -version IDs for existing unversioned objects. MinIO lists unversioned -objects with a ``null`` version ID. + .. card:: Retrieving a Specific Object Version + + .. image:: /images/retention/minio-versioning-retrieve-single-version.svg + :alt: Object with Multiple Versions + :align: center + +:ref:`Deleting ` an object results in a special +``DeleteMarker`` tombstone that marks an object as deleted while retaining +all previous versions of that object. + +Versioning is Per-Namespace +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses the full namespace (the bucket and path to an object) for each object +as part of determining object uniqueness. For example, all of the following +namespaces are "unique" objects, where mutations of each object result in +the creation of new object versions *at that namespace*: + +.. code-block:: shell + + databucket/object.blob + databucket/blobs/object.blob + blobbucket/object.blob + blobbucket/blobs/object.blob + +While ``object.blob`` might be the same binary across all namespaces, +MinIO only enforces versioning with a specific namespace and therefore +considers each ``object.blob`` above as distinct and unique. + +Versioning and Storage Capacity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO does not perform incremental or differential-type versioning. For +mutation-heavy workloads, this may result in substantial disk usage by +older or aged object versions. + +For example, consider a 1GB object containing log data. An application +appends 100MB of data to the log and uploads to MinIO. MinIO would then contain +both the 1GB and 1.1GB versions of the object. If the application repeated +this process every day for 10 days, the bucket would eventually contain more +than 14GB of data associated to a single object. + +MinIO supports configuring configuring :ref:`object lifecycle management rules +` to automatically expire or transition aged +object versions and free up storage capacity. For example, you can configure +a rule to automatically expire object versions 90 days after they become +non-current (i.e. no longer the "latest" version of that object). See +:ref:`MinIO Object Expiration ` for +more information. + +You can alternatively perform manual removal of object versions using the +following commands: + +- :mc-cmd-option:`mc rm versions` - Removes all versions of an object. +- :mc-cmd-option:`mc rm --versions --older-than ` - + Removes all versions of an object older than the specified calendar date. + +.. _minio-bucket-versioning-id: Version ID Generation ---------------------- +~~~~~~~~~~~~~~~~~~~~~ -MinIO version ID's are DCE 1.1/RFC 4122-compliant Universally Unique Identifiers -(UUID) version 4. Each UUID is a random 128-bit number intended to have a high -likelihood of uniqueness over space and time, *and* that are computationally -difficult to guess. UUID's are globally unique that can be generated without -contacting a global registration authority. +MinIO generates a unique and immutable identifier for each versioned object as +part of write operations. Each object version ID consists of a 128-bit +fixed-size :rfc:`UUIDv4 <4122#section-4.4>`. UUID generation is sufficiently +random to ensure high likelihood of uniqueness for any environment, are +computationally difficult to guess, and do not require centralized registration +process and authority to guarantee uniqueness. -MinIO object version UUID's are *immutable* after creation. +.. image:: /images/retention/minio-versioning-multiple-versions.svg + :alt: Object with Multiple Versions + :width: 600px + :align: center -Versioning Dependent Features ------------------------------ +MinIO does not support client-managed version ID allocation. All version ID +generation is handled by the MinIO server process. -The following MinIO features *require* bucket versioning for functionality: +For objects created while versioning is disabled or suspended, MinIO +uses a ``null`` version ID. You can access or remove these objects by specifying +``null`` as the version ID as part of S3 operations. + +.. _minio-bucket-versioning-delete: + +Versioned Delete Operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Performing a ``DELETE`` operation on a versioned object creates a +0-byte ``DeleteMarker`` as the latest version of that object. Clients performing +``GET`` operations on that object do not return any results, as MinIO does not +return the ``DeleteMarker`` back as part of the response. Similarly, performing +a ``LIST`` operation by default returns only objects which are *not* a +``DeleteMarker``. + +To permanently delete an object version, perform the ``DELETE`` operation and +specify the version ID of the object to delete. Versioned delete operations +are **irreversible**. + +.. card-carousel:: 1 + + .. card:: Deleting an Object + + .. image:: /images/retention/minio-versioning-delete-object.svg + :alt: Deleting an Object + :align: center + + Performing a ``DELETE`` operation on a versioned object produces a + ``DeleteMarker`` for that object. + + .. card:: Reading a Deleted Object + + .. image:: /images/retention/minio-versioning-retrieve-deleted-object.svg + :alt: Object with Multiple Versions + :align: center + + Clients by default retrieve the "latest" object version. MinIO returns + a ``404``-like response if the latest version is a ``DeleteMarker``. + + .. card:: Retrieve Previous Version of Deleted Object + + .. image:: /images/retention/minio-versioning-retrieve-version-before-delete.svg + :alt: Retrieve Version of Deleted Object + :align: center + + Clients can retrieve any previous version of the object by specifying the + version ID, even if the "Latest" version is a ``DeleteMarker``. + + .. card:: Delete a Specific Object Version + + .. image:: /images/retention/minio-versioning-delete-specific-version.svg + :alt: Retrieve Version of Deleted Object + :align: center + + Clients can delete a specific object version by specifying the version ID + as part of the ``DELETE`` operation. Deleting a specific version is + **permanent** and does not result in the creation of a ``DeleteMarker``. + +The following :mc:`mc` commands operate on ``DeleteMarkers`` or versioned +objects: + +- Use :mc-cmd-option:`mc ls versions` to view all versions of an object, + including delete markers. + +- Use :mc-cmd-option:`mc cp --version-id=UUID ... ` to + retrieve the version of the "deleted" object with matching ``UUID``. + +- Use :mc-cmd-option:`mc rm --version-id=UUID ... ` to delete + the version of the object with matching ``UUID``. + +- Use :mc-cmd-option:`mc rm versions` to delete *all* versions of an object. + +Tutorials +--------- + +Enable Bucket Versioning +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + Versioning requires :ref:`erasure coding ` and is only + available on :ref:`distributed MinIO deployments `. + +You can enable versioning using the MinIO Console, the MinIO :mc:`mc` CLI, or +using an S3-compatible SDK. Versioning is a bucket-scoped feature. You cannot +enable versioning on only a prefix or subset of objects in a bucket. + +.. tab-set:: + + .. tab-item:: MinIO Console + + Select the :guilabel:`Buckets` section of the MinIO Console to access + bucket creation and management functions. Select the bucket row from the + list of buckets. You can use the :octicon:`search` :guilabel:`Search` bar + to filter the list. + + .. image:: /images/minio-console/console-bucket.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + From the Bucket view, click the :guilabel:`Versioning` + field to enable versioning on the bucket. + + .. image:: /images/minio-console/console-bucket-overview.png + :width: 600px + :alt: MinIO Console Bucket minio-lifecycle-management + :align: center + + The MinIO Console also supports enabling versioning as part of bucket + creation. See :ref:`minio-console-admin-buckets` for more information on + bucket management using the MinIO Console. + + .. tab-item:: MinIO CLI + + Use the :mc-cmd:`mc version enable` command to enable versioning on an + existing bucket: + + .. code-block:: shell + :class: copyable + + mc version enable ALIAS/BUCKET + + - Replace ``ALIAS`` with the :mc:`alias ` of a configured + MinIO deployment. + + - Replace ``BUCKET`` with the + :mc:`target bucket ` on which to enable + versioning. + +Objects created prior to enabling versioning have a +``null`` :ref:`version ID `. + +Suspend Bucket Versioning +~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can suspend bucket versioning at any time using the MinIO Console, the +MinIO :mc:`mc` CLI, or using an S3-compatible SDK. + +.. tab-set:: + + .. tab-item:: MinIO Console + + Select the :guilabel:`Buckets` section of the MinIO Console to access + bucket creation and management functions. + + Select the bucket row from the list of buckets. You can use the + :octicon:`search` :guilabel:`Search` bar to filter the list. + + .. image:: /images/minio-console/console-bucket.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + From the :guilabel:`Bucket` view, click the :guilabel:`Versioning` field + to disable versioning on the bucket. + + .. image:: /images/minio-console/console-bucket-overview.png + :width: 600px + :alt: MinIO Console Bucket minio-lifecycle-management + :align: center + + See :ref:`minio-console-admin-buckets` for more information on bucket + management using the MinIO Console. + + .. tab-item:: MinIO CLI + + Use the :mc-cmd:`mc version disable` command to enable versioning on an + existing bucket: + + .. code-block:: shell + :class: copyable + + mc version disable ALIAS/BUCKET + + - Replace ``ALIAS`` with the :mc:`alias ` of a configured + MinIO deployment. + + - Replace ``BUCKET`` with the + :mc:`target bucket ` on which to disable + versioning. + +Objects created while versioning is suspended are assigned a +``null`` :ref:`version ID `. Any mutations to an +object while versioning is suspended results in overwriting that +``null`` versioned object. MinIO does not remove or otherwise alter +existing versioned objects as part of suspending versioning. Clients can +continue interacting with any existing object versions in the bucket. -- Object Locking (:mc-cmd:`mc lock` and :mc-cmd-option:`mc mb with-lock`) -- Object Legal Hold (:mc-cmd:`mc legalhold`) -- Bucket Replication (:mc-cmd:`mc admin bucket remote` and :mc-cmd:`mc replicate`) \ No newline at end of file diff --git a/source/concepts/feature-overview.rst b/source/concepts/feature-overview.rst index fd65b74c..cf15f566 100644 --- a/source/concepts/feature-overview.rst +++ b/source/concepts/feature-overview.rst @@ -44,5 +44,4 @@ The following table lists MinIO features and their corresponding documentation: :titlesonly: :hidden: - /concepts/bucket-versioning /concepts/erasure-coding diff --git a/source/console/minio-console.rst b/source/console/minio-console.rst index df7630a8..44184847 100644 --- a/source/console/minio-console.rst +++ b/source/console/minio-console.rst @@ -194,6 +194,8 @@ change or retrieve the credentials later. To rotate credentials for an application, create a new service account and delete the old one once the application updates to using the new credentials. +.. _minio-console-admin-buckets: + Admin: Buckets -------------- @@ -216,7 +218,7 @@ You can select a bucket to view more specific details for that bucket: .. image:: /images/minio-console/console-bucket-overview.png :width: 600px - :alt: MinIO Console Create Bucket + :alt: MinIO Console Bucket Management :align: center - The :guilabel:`Summary` tab displays a summary of the bucket configuration. diff --git a/source/images/minio-console/console-bucket-locking-compliance.png b/source/images/minio-console/console-bucket-locking-compliance.png new file mode 100644 index 00000000..0366ff84 Binary files /dev/null and b/source/images/minio-console/console-bucket-locking-compliance.png differ diff --git a/source/images/minio-console/console-bucket-locking.png b/source/images/minio-console/console-bucket-locking.png new file mode 100644 index 00000000..4435064c Binary files /dev/null and b/source/images/minio-console/console-bucket-locking.png differ diff --git a/source/images/minio-console/console-bucket-overview.png b/source/images/minio-console/console-bucket-overview.png index c6f29b07..cf780a79 100644 Binary files a/source/images/minio-console/console-bucket-overview.png and b/source/images/minio-console/console-bucket-overview.png differ diff --git a/source/images/minio-console/console-object-browser-locking.png b/source/images/minio-console/console-object-browser-locking.png new file mode 100644 index 00000000..98ef11f3 Binary files /dev/null and b/source/images/minio-console/console-object-browser-locking.png differ diff --git a/source/images/minio-console/console-object-browser-object-details.png b/source/images/minio-console/console-object-browser-object-details.png new file mode 100644 index 00000000..d06cf092 Binary files /dev/null and b/source/images/minio-console/console-object-browser-object-details.png differ diff --git a/source/images/retention/minio-object-locking-delete-version.svg b/source/images/retention/minio-object-locking-delete-version.svg new file mode 100644 index 00000000..e7854148 --- /dev/null +++ b/source/images/retention/minio-object-locking-delete-version.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-object-locking-delete.svg b/source/images/retention/minio-object-locking-delete.svg new file mode 100644 index 00000000..846d1b49 --- /dev/null +++ b/source/images/retention/minio-object-locking-delete.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-object-locking.svg b/source/images/retention/minio-object-locking.svg new file mode 100644 index 00000000..37e4c67e --- /dev/null +++ b/source/images/retention/minio-object-locking.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-default.svg b/source/images/retention/minio-versioning-default.svg new file mode 100644 index 00000000..027a3fa2 --- /dev/null +++ b/source/images/retention/minio-versioning-default.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-delete-object.svg b/source/images/retention/minio-versioning-delete-object.svg new file mode 100644 index 00000000..a8c36679 --- /dev/null +++ b/source/images/retention/minio-versioning-delete-object.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-delete-specific-version.svg b/source/images/retention/minio-versioning-delete-specific-version.svg new file mode 100644 index 00000000..c5421698 --- /dev/null +++ b/source/images/retention/minio-versioning-delete-specific-version.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-multiple-versions.svg b/source/images/retention/minio-versioning-multiple-versions.svg new file mode 100644 index 00000000..b87ec5fe --- /dev/null +++ b/source/images/retention/minio-versioning-multiple-versions.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-retrieve-deleted-object.svg b/source/images/retention/minio-versioning-retrieve-deleted-object.svg new file mode 100644 index 00000000..a14af2a8 --- /dev/null +++ b/source/images/retention/minio-versioning-retrieve-deleted-object.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-retrieve-latest-version.svg b/source/images/retention/minio-versioning-retrieve-latest-version.svg new file mode 100644 index 00000000..d8fcdc83 --- /dev/null +++ b/source/images/retention/minio-versioning-retrieve-latest-version.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-retrieve-single-version.svg b/source/images/retention/minio-versioning-retrieve-single-version.svg new file mode 100644 index 00000000..379c36a8 --- /dev/null +++ b/source/images/retention/minio-versioning-retrieve-single-version.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-retrieve-version-before-delete.svg b/source/images/retention/minio-versioning-retrieve-version-before-delete.svg new file mode 100644 index 00000000..de37f93d --- /dev/null +++ b/source/images/retention/minio-versioning-retrieve-version-before-delete.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/images/retention/minio-versioning-single-version.svg b/source/images/retention/minio-versioning-single-version.svg new file mode 100644 index 00000000..5755847a --- /dev/null +++ b/source/images/retention/minio-versioning-single-version.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/source/index.rst b/source/index.rst index ee31bd20..d5c4b60e 100644 --- a/source/index.rst +++ b/source/index.rst @@ -134,6 +134,7 @@ such as versioning and replication. /introduction/minio-overview /concepts/feature-overview + Object Retention /installation/deployment-and-management /lifecycle-management/lifecycle-management-overview /replication/replication-overview diff --git a/source/object-retention/minio-object-locking.rst b/source/object-retention/minio-object-locking.rst new file mode 100644 index 00000000..25de786b --- /dev/null +++ b/source/object-retention/minio-object-locking.rst @@ -0,0 +1,446 @@ +.. _minio-object-locking: + +==================== +MinIO Object Locking +==================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +Overview +-------- + +MinIO Object Locking ("Object Retention") enforces Write-Once Read-Many (WORM) +immutability to protect :ref:`versioned objects ` from +deletion. MinIO supports both +:ref:`duration based object retention ` +and +:ref:`indefinite Legal Hold retention `. + +MinIO Object Locking provides key data retention compliance and meets +SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d) requirements as per +`Cohasset Associates `__. + +.. card-carousel:: 1 + + .. card:: Bucket Without Locking + + .. image:: /images/retention/minio-versioning-delete-object.svg + :alt: Deleting an Object + :align: center + + MinIO versioning preserves the full history of object mutations. + However, applications can explicitly delete specific object versions. + + .. card:: Bucket With Locking + + .. image:: /images/retention/minio-object-locking.svg + :alt: 30 Day Locked Objects + :align: center + + Applying a default 30 Day WORM lock to objects in the bucket ensures + a minimum period of retention and protection for all object versions. + + .. card:: Delete Operations in Locked Bucket + + .. image:: /images/retention/minio-object-locking-delete.svg + :alt: Delete Operation in Locked Bucket + :align: center + + Delete operations follow normal behavior in + :ref:`versioned buckets `, where MinIO + creates a ``DeleteMarker`` for the object. However, non-Delete Marker + versions of the object remain under the retention rules and are protected + from any specific deletion or overwrite attempts. + + .. card:: Versioned Delete Operations in Locked Bucket + + .. image:: /images/retention/minio-object-locking-delete-version.svg + :alt: Versioned Delete Operation in a Locked Bucket + :align: center + + MinIO blocks any attempt to delete a specific object version held under + WORM lock. The earliest possible time after which a client may delete + the version is when the lock expires. + +MinIO object locking is +:s3-docs:`feature and API compatible with AWS S3 `. +This page summarizes Object Locking / Retention concepts as implemented by +MinIO. See the AWS S3 documentation on +:s3-docs:`How S3 Object Lock works ` for additional +resources. + +You can only enable object locking during bucket creation as per +:s3-docs:`S3 behavior `. +You cannot enable object locking on a bucket created without locking +enabled. You can then configure object retention rules at any time. +Object locking requires :ref:`versioning ` and +enables the feature implicitly. + +.. _minio-bucket-locking-interactions-versioning: + +Interaction with Versioning +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Objects held under WORM locked are immutable until the lock expires or is +explicitly lifted. Locking is per-object version, where each version is +independently immutable. + +If an application performs an unversioned delete operation on a locked object, +the operation produces a :ref:`delete marker `. +Attempts to explicitly delete any WORM-locked object fail with an error. +Delete Markers are *not* eligible for protection under WORM locking. +See the S3 documentation on +:s3-docs:`Managing delete markers and object lifecycles +` for more +information. + +For example, consider the following bucket with +:ref:`minio-object-locking-governance` locking enabled by default: + +.. code-block:: shell + + $ mc ls --versions play/locking-guide + + [DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv + [DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv + [DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv + +Attempting to perform a delete on a *specific version* of ``data.csv`` fails +due to the object locking settings: + +.. code-block:: shell + + $ mc rm --version-id 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 play/data.csv + + Removing `play/locking-guide/data.csv` (versionId=62429eb1-9cb7-4dc5-b507-9cc23d0cc691). + mc: Failed to remove `play/locking-guide/data.csv`. + Object, 'data.csv (Version ID=62429eb1-9cb7-4dc5-b507-9cc23d0cc691)' is + WORM protected and cannot be overwritten + +Attempting to perform an unversioned delete on ``data.csv`` succeeds and creates +a new ``DeleteMarker`` for the object: + +.. code-block:: shell + + $ mc rm play/locking-guide/data.csv + + [DATETIME] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv + [DATETIME] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv + [DATETIME] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv + [DATETIME] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv + +Interaction with Lifecycle Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO :ref:`object expiration ` +respects any active object lock and retention settings for objects covered by +the expiration rule. + +- For expiration rules operating on only the *current* object version, + MinIO creates a Delete Marker for the locked object. + +- For expiration rules operating on *non-current object versions*, + MinIO can only expire the non-current versions *after* the retention period + has passed *or* has been explicitly lifted (e.g. Legal Holds). + +For example, consider the following bucket with +:ref:`minio-object-locking-governance` locking enabled by default for 45 days: + +.. code-block:: shell + + $ mc ls --versions play/locking-guide + + [7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv + [30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv + [60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv + +Creating an expiration rule for *current* objects older than 7 days results in +a Delete Marker for the object: + +.. code-block:: shell + + $ mc ls --versions play/locking-guide + + [0D] 0B acce329f-ad32-46d9-8649-5fe8bf4ec6e0 v4 DEL data.csv + [7D] 29B 62429eb1-9cb7-4dc5-b507-9cc23d0cc691 v3 PUT data.csv + [30D] 32B 78b3105a-02a1-4763-8054-e66add087710 v2 PUT data.csv + [60D] 23B c6b581ca-2883-41e2-9905-0a1867b535b8 v1 PUT data.csv + +However, an expiration rule for *non-current* objects older than 7 days would +only take effect *after* the configured WORM lock expires. Since the bucket +has a 45 day ``GOVERNANCE`` retention set, only the ``v1`` version of +``data.csv`` is unlocked and therefore eligible for deletion. + +Tutorials +--------- + +Create Bucket with Object Locking Enabled +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You must enable object locking during bucket creation as per S3 behavior. +You can create a bucket with object locking enabled using the MinIO Console, +the MinIO :mc:`mc` CLI, or using an S3-compatible SDK. + +.. tab-set:: + + .. tab-item:: MinIO Console + :sync: console + + Select the :guilabel:`Buckets` section of the MinIO Console to access + bucket creation and management functions. Select the bucket row from the + list of buckets. You can use the :octicon:`search` :guilabel:`Search` bar + to filter the list. + + .. image:: /images/minio-console/console-bucket.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + Click the :guilabel:`Create Bucket` button to open the bucket creation + model. Toggle the :guilabel:`Object Locking` selector to enable object + locking on the bucket. + + .. image:: /images/minio-console/console-bucket-create-bucket.png + :width: 600px + :alt: MinIO Console Bucket creation + :align: center + + .. tab-item:: MinIO CLI + :sync: cli + + Use the :mc-cmd:`mc mb` command with the :mc-cmd-option:`~mc mb with-lock` + option to create a bucket with object locking enabled: + + .. code-block:: shell + :class: copyable + + mc mb --with-lock ALIAS/BUCKET + + - Replace ``ALIAS`` with the :mc:`alias ` of a configured + MinIO deployment. + + - Replace ``BUCKET`` with the + :mc:`name ` of the bucket to create. + +Configure Bucket-Default Object Retention +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can configure object locking rules ("object retention") using the +MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK. + +MinIO supports setting both bucket-default *and* per-object retention rules. +The following examples set bucket-default retention. For per-object retention +settings, defer to the documentation for the ``PUT`` operation used by your +preferred SDK. + +.. tab-set:: + + .. tab-item:: MinIO Console + :sync: console + + Select the :guilabel:`Buckets` section of the MinIO Console to access + bucket creation and management functions. Select the bucket row from the + list of buckets. You can use the :octicon:`search` :guilabel:`Search` bar + to filter the list. + + .. image:: /images/minio-console/console-bucket.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + From the :guilabel:`Bucket` view, look for the + :guilabel:`Retention` section and click :guilabel:`Enabled`. This section + is only visible if the bucket was created with object locking enabled. + + .. image:: /images/minio-console/console-bucket-overview.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + From the :guilabel:`Set Retention Configuration` modal, set the + desired bucket default retention settings. + + .. image:: /images/minio-console/console-bucket-locking-compliance.png + :width: 400px + :alt: MinIO Console Bucket Default Retention + :align: center + + - For :guilabel:`Retention Mode`, select either + :ref:`COMPLIANCE ` or + :ref:`GOVERNANCE `. + + - For :guilabel:`Duration`, select the retention duration units of + :guilabel:`Days` or :guilabel:`Years`. + + - For :guilabel:`Retention Validity`, set the duration of time for which + MinIO holds objects under the specified retention mode for the bucket. + + .. tab-item:: MinIO CLI + :sync: cli + + Use the :mc-cmd:`mc retention` command with the + :mc-cmd-option:`--recursive ` and + :mc-cmd-option:`--default ` options to set the + default retention mode for a bucket: + + .. code-block:: shell + :class: copyable + + mc retention set --recursive --default MODE DURATION ALIAS/BUCKET + + - Replace :mc-cmd:`MODE ` with either either + :ref:`COMPLIANCE ` or + :ref:`GOVERNANCE `. + + - Replace :mc-cmd:`DURATION ` with the + duration for which the object lock remains in effect. + + - Replace :mc-cmd:`ALIAS ` with the + :mc:`alias ` of a configured MinIO deployment. + + - Replace :mc-cmd:`BUCKET ` with the + name of the bucket on which to set the default retention rule. + +Enable Legal Hold Retention +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can enable or disable indefinite Legal Hold retention for an object using +the MinIO Console, the MinIO :mc:`mc` CLI, or using an S3-compatible SDK. +You can place a legal hold on an object already held under a +:ref:`COMPLIANCE ` or +:ref:`GOVERNANCE ` lock. The object remains +WORM locked until the retention lock expires *and* the legal hold is lifted. + +.. tab-set:: + + .. tab-item:: MinIO Console + :sync: console + + Select the :guilabel:`Object Browser` section of the MinIO Console. Select + the bucket row from the list of buckets. You can use the :octicon:`search` + :guilabel:`Search` bar to filter the list. + + .. image:: /images/minio-console/console-object-browser-locking.png + :width: 600px + :alt: MinIO Console Bucket Management + :align: center + + Browse to the object and select it to open the object details view. + Click the :octicon:`pencil` icon on the :guilabel:`Legal Hold` row to + toggle the Legal Hold status of the object. + + .. image:: /images/minio-console/console-object-browser-object-details.png + :width: 600px + :alt: MinIO Console Bucket Default Retention + :align: center + + .. tab-item:: MinIO CLI + :sync: cli + + Use the :mc-cmd:`mc legalhold` command to enable or disable the legal + hold on an object. + + .. code-block:: shell + :class: copyable + + mc legalhold set ALIAS/PATH + + - Replace :mc-cmd:`ALIAS ` with the + :mc:`alias ` of a configured MinIO deployment. + + - Replace :mc-cmd:`PATH ` with the + path to the object for which to enable the legal hold. + +.. _minio-object-locking-retention-modes: + +Object Retention Modes +---------------------- + +MinIO implements the following +:s3-docs:`S3 Object Locking Modes `: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + :width: 100% + + * - Mode + - Summary + + * - :ref:`minio-object-locking-governance` + - Prevents any operation that would mutate or modify the object or its + locking settings by non-privileged users. + + Users with the :policy-action:`s3:BypassGovernanceRetention` permission + on the bucket or object can modify the object or its locking settings. + + MinIO lifts the lock automatically after the configured retention rule + duration has passed. + + * - :ref:`minio-object-locking-compliance` + - Prevents any operation that would mutate or modify the object or its + locking settings. + + No MinIO user can modify the object or its settings, including the + :ref:`MinIO root ` user. + + MinIO lifts the lock automatically after the configured retention rule + duration has passed. + +.. _minio-object-locking-governance: + +GOVERNANCE Mode +~~~~~~~~~~~~~~~ + +An object under ``GOVERNANCE`` lock is protected from write operations by +non-privileged users. + +``GOVERNANCE`` locked objects enforce managed-immutability for locked objects, +where users with the :policy-action:`s3:BypassGovernanceRetention` action can +modify the locked object, change the retention duration, or lift the lock +entirely. Bypassing ``GOVERNANCE`` retention also requires setting the +``x-amz-bypass-governance-retention:true`` header as part of the request. + +The MinIO ``GOVERNANCE`` lock is functionally identical to the +:s3-docs:`S3 GOVERNANCE mode +`. + +.. _minio-object-locking-compliance: + +COMPLIANCE Mode +~~~~~~~~~~~~~~~ + +An object under ``COMPLIANCE`` lock is protected from write operations by *all* +users, including the :ref:`MinIO root ` user. + +``COMPLIANCE`` locked objects enforce complete immutability for locked objects. +However, you cannot change or remove the lock before the configured retention +duration has passed. + +The MinIO ``COMPLIANCE`` lock is functionally identical to the +:s3-docs:`S3 GOVERNANCE mode +`. + +.. _minio-object-locking-legalhold: + +Legal Hold +---------- + +An object under Legal Hold is protected from write operations by *all* +users, including the :ref:`MinIO root ` user. + +Legal Holds are indefinite and enforce complete immutability for locked objects. +Only privileged users with the :policy-action:`s3:PutObjectLegalHold` can set or +lift the Legal Hold. + +Legal holds are complementary to both :ref:`minio-object-locking-governance` and +:ref:`minio-object-locking-compliance` retention settings. An object held under +both legal hold *and* a ``GOVERNANCE/COMPLIANCE`` retention rule remains WORM +locked until the legal hold is lifed *and* the rule expires. + +For ``GOVERNANCE`` locked objects, the legal hold prevents mutating the object +*even if* the user has the necessary privileges to bypass retention. \ No newline at end of file diff --git a/source/object-retention/minio-object-retention.rst b/source/object-retention/minio-object-retention.rst new file mode 100644 index 00000000..75bba5af --- /dev/null +++ b/source/object-retention/minio-object-retention.rst @@ -0,0 +1,60 @@ +.. _minio-object-retention: + +====================== +MinIO Object Retention +====================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + +By default, each new write operation to a unique object name results in +overwriting that object. You can configure MinIO to instead create +:ref:`versions ` of each object mutation, preserving +the full history of that object. MinIO also supports +:ref:`Write-Once Read-Many (WORM) locking ` versioned +objects to ensure complete immutability for a specified duration *or* until the +lock is explicitly lifted. + +Both versioning and object locking features are available only with +:ref:`distributed MinIO deployments `. + +.. card:: Bucket Versioning + :link: minio-bucket-versioning + :link-type: ref + + MinIO keeps each mutation to an object as a full "version" of that object. + + .. image:: /images/retention/minio-versioning-multiple-versions.svg + :width: 600px + :alt: Bucket with multiple versions of an object + :align: center + + Clients by default retrive the latest version of an object and can + explicitly list and retrieve any other version in the object's history. + +.. card:: WORM Object Locking + :link: minio-object-locking + :link-type: ref + + MinIO enforces Write-Once Read Many (WORM) immutability on versioned objects. + Clients cannot delete a WORM-locked object until the configured locking + rules expire or are explicitly lifted. + + .. image:: /images/retention/minio-object-locking.svg + :width: 600px + :alt: Bucket with multiple versions of an object + :align: center + + MinIO supports setting bucket-default or per-object WORM locking rules with + either duration-based or indefinite lock expirations. MinIO object locking is + feature-compatible with AWS S3 object locking. + +.. toctree:: + :titlesonly: + :hidden: + + Object Versioning + Object Locking \ No newline at end of file diff --git a/source/reference/minio-cli/minio-mc/mc-retention.rst b/source/reference/minio-cli/minio-mc/mc-retention.rst index 7bd16001..022ac6fc 100644 --- a/source/reference/minio-cli/minio-mc/mc-retention.rst +++ b/source/reference/minio-cli/minio-mc/mc-retention.rst @@ -17,13 +17,16 @@ Description .. start-mc-retention-desc -The :mc:`mc retention` command configures the Write-Once Read-Many (WORM) object -lock settings for an object or object(s) in a bucket. You can also set the -default object lock settings for a bucket, where all objects without explicit -object lock settings inherit the bucket default. +The :mc:`mc retention` command configures the :ref:`Write-Once Read-Many (WORM) +locking ` settings for an object or object(s) in a bucket. +You can also set the default object lock settings for a bucket, where all +objects without explicit object lock settings inherit the bucket default. .. end-mc-retention-desc +To lock an object under :ref:`legal hold `, +use :mc:`mc legalhold`. + .. note:: Starting in version :mc-release:`RELEASE.2020-09-18T00-13-21Z`, diff --git a/source/replication/enable-server-side-one-way-bucket-replication.rst b/source/replication/enable-server-side-one-way-bucket-replication.rst index 818ce0cb..18c81bc5 100644 --- a/source/replication/enable-server-side-one-way-bucket-replication.rst +++ b/source/replication/enable-server-side-one-way-bucket-replication.rst @@ -213,6 +213,19 @@ destination clusters to ensure the safety of objects during transmission. MinIO does *not* support replicating client-side encrypted objects (SSE-C). +Replication of Locked Objects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO supports replicating objects held under +:ref:`WORM Locking `. Both the source and destination +buckets *must* have object locking enabled for MinIO to replicate the locked +object. + +You must enable object locking during bucket creation as per S3 behavior. +You can then configure object retention rules at any time. +Object locking requires :ref:`versioning ` and +enables the feature implicitly. + Procedure --------- diff --git a/source/replication/enable-server-side-two-way-bucket-replication.rst b/source/replication/enable-server-side-two-way-bucket-replication.rst index 1e92caf1..aa1310ad 100644 --- a/source/replication/enable-server-side-two-way-bucket-replication.rst +++ b/source/replication/enable-server-side-two-way-bucket-replication.rst @@ -212,13 +212,20 @@ destination clusters to ensure the safety of objects during transmission. MinIO does *not* support replicating client-side encrypted objects (SSE-C). -Replication of Object Retention (WORM, Legal Hold) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Replication of Locked Objects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -MinIO supports replicating objects with retention settings, such as -:abbr:`WORM (Write-Once Read-Many)` object locking or legal holds. Both the -source and destination bucket *must* have object locking enabled for MinIO -to replicate objects with their associated retention settings. +MinIO supports replicating objects held under +:ref:`WORM Locking `. Both replication buckets *must* have +object locking enabled for MinIO to replicate the locked object. For +active-active configuration, MinIO recommends using the *same* +retention rules on both buckets to ensure consistent behavior across +sites. + +You must enable object locking during bucket creation as per S3 behavior. +You can then configure object retention rules at any time. +Object locking requires :ref:`versioning ` and +enables the feature implicitly. Procedure ---------