From 31d5fbd1170ec9f46682b9398b93cac8565fb872 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Tue, 28 Mar 2023 16:51:11 -0600 Subject: [PATCH] Exclude prefixes or folders from versioning (#769) Object versioning can now be configured with certain exclusions (no longer strictly on a per-bucket basis). Add documentation about how to use the new options: * `--excluded-prefixes` to exclude objects with certain prefixes * `--exclude-folders` to exclude anything with name ending in `/` (folders) - [x] Exclude prefixes - [x] Exclude folders - [x] Wildcards/globbing - [x] Limitations - [x] View versioning exclusion configuration Staged: [Bucket Versioning](http://192.241.195.202:9000/staging/DOCS-760/linux/html/administration/object-management/object-versioning.html) [mc version](http://192.241.195.202:9000/staging/DOCS-760/linux/html/reference/minio-mc/mc-version.html) Fixes https://github.com/minio/docs/issues/760 --------- Co-authored-by: Krishnan Parthasarathi --- source/administration/concepts.rst | 2 + .../object-management/object-versioning.rst | 126 +++++++++++++++++- source/reference/minio-mc/mc-version.rst | 47 +++++-- 3 files changed, 161 insertions(+), 14 deletions(-) diff --git a/source/administration/concepts.rst b/source/administration/concepts.rst index 4fdded2a..d8ce7f5a 100644 --- a/source/administration/concepts.rst +++ b/source/administration/concepts.rst @@ -67,6 +67,8 @@ Once authenticated, TLS provides the cipher to encrypt and then decrypt the tran MinIO supports several methods of :ref:`Server-Side Encryption `. +.. _minio-admin-concepts-organize-objects: + Can I organize objects in a folder structure within buckets? ------------------------------------------------------------ diff --git a/source/administration/object-management/object-versioning.rst b/source/administration/object-management/object-versioning.rst index 1e71b247..421df527 100644 --- a/source/administration/object-management/object-versioning.rst +++ b/source/administration/object-management/object-versioning.rst @@ -219,8 +219,7 @@ Enable Bucket Versioning ~~~~~~~~~~~~~~~~~~~~~~~~ You can enable versioning using the MinIO Console, the MinIO :mc:`mc` CLI, or -using an S3-compatible SDK. Versioning is a bucket-scoped feature. You cannot -enable versioning on only a prefix or subset of objects in a bucket. +using an S3-compatible SDK. .. tab-set:: @@ -266,6 +265,129 @@ enable versioning on only a prefix or subset of objects in a bucket. Objects created prior to enabling versioning have a ``null`` :ref:`version ID `. + +Exclude a Prefix From Versioning +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can exclude certain :ref:`prefixes ` from versioning using the :ref:`MinIO Client `. +This is useful for Spark/Hadoop workloads or others that initially create objects with temporary prefixes. + +.. admonition:: Object locking + :class: note + + Buckets with :ref:`object locking enabled ` require versioning and do not support excluding prefixes. + +- Use :mc-cmd:`mc version enable` with the :mc-cmd:`~mc version --excluded-prefixes` option: + + .. code-block:: shell + :class: copyable + + mc version enable --excluded-prefixes "prefix1, prefix2" ALIAS/BUCKET + + - Replace ``ALIAS`` with the :mc:`alias ` of a configured MinIO deployment. + + - Replace ``BUCKET`` with the name of the :s3-docs:`bucket ` you want to exclude :ref:`prefixes ` for. + +The list of :mc-cmd:`~mc version --excluded-prefixes` prefixes match all objects containing the specified strings in their prefix or name, similar to a regular expression of the form ``prefix*``. +To match objects by prefix only, use ``prefix/*``. + +For example, the following command excludes any objects containing ``_test`` or ``_temp`` in their prefix or name from versioning: + + .. code-block:: shell + :class: copyable + + mc version enable --excluded-prefixes "_test, _temp" local/my-bucket + +You can exclude up to 10 prefixes for each bucket. +To add or remove prefixes, repeat the :mc-cmd:`mc version enable` command with an updated list. +The new list of prefixes replaces the previous one. + +To view the currently excluded prefixes, use :mc-cmd:`mc version info` with the :mc-cmd:`~mc version enable --JSON` option: + + .. code-block:: shell + :class: copyable + + mc version info ALIAS/BUCKET --json + +The command output resembles the following, with the list of excluded prefixes in the ``ExcludedPrefixes`` property: + +.. code-block:: shell + + $ mc version info local/my-bucket --json + { + "Op": "info", + "status": "success", + "url": "local/my-bucket", + "versioning": { + "status": "Enabled", + "MFADelete": "", + "ExcludedPrefixes": [ + "prefix1, prefix2" + ] + } + } + +To disable prefix exclusion and resume versioning all prefixes, repeat the :mc-cmd:`mc version enable` command without :mc-cmd:`~mc version --excluded-prefixes`: + + .. code-block:: shell + :class: copyable + + mc version enable ALIAS/BUCKET + + +Exclude Folders from Versioning +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can exclude folders from versioning using the :ref:`MinIO Client `. + +.. admonition:: Object locking + :class: note + + Buckets with :ref:`object locking enabled ` require versioning and do not support excluding folders. + +- Use :mc-cmd:`mc version enable` with the :mc-cmd:`~mc version --exclude-folders` option to exclude objects with names ending in ``/`` from versioning: + + .. code-block:: shell + :class: copyable + + mc version enable --exclude-folders ALIAS/BUCKET + + - Replace ``ALIAS`` with the :mc:`alias ` of a configured MinIO deployment. + + - Replace ``BUCKET`` with the :s3-docs:`bucket ` you want to exclude :ref:`folders ` for. + +To check whether folders are versioned for a bucket, use the :mc-cmd:`mc version enable` command with the ``--json`` option. +If the ``ExcludeFolders`` property is ``true``, folders in that bucket are not versioned. + + .. code-block:: shell + :class: copyable + + mc version enable --excluded-prefixes ALIAS/BUCKET --json + +The command output resembles the following: + +.. code-block:: shell + + $ mc version info local/my-bucket --json + { + "Op": "info", + "status": "success", + "url": "local/my-bucket", + "versioning": { + "status": "Enabled", + "MFADelete": "", + "ExcludeFolders": true + } + } + +To disable folder exclusion and resume versioning all folders, repeat the :mc-cmd:`mc version enable` command without :mc-cmd:`~mc version --exclude-folders`: + + .. code-block:: shell + :class: copyable + + mc version enable ALIAS/BUCKET + + Suspend Bucket Versioning ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/minio-mc/mc-version.rst b/source/reference/minio-mc/mc-version.rst index 07168cfe..3175910f 100644 --- a/source/reference/minio-mc/mc-version.rst +++ b/source/reference/minio-mc/mc-version.rst @@ -52,7 +52,18 @@ The :mc:`mc version` command enables, suspends, and retrieves the Parameters ~~~~~~~~~~ +.. mc-cmd:: ALIAS + :required: + + The :ref:`alias ` of a MinIO deployment and the full path + to the bucket for which to set the versioning configuration. For example: + + .. code-block:: shell + + mc version enable myminio/mybucket + .. mc-cmd:: enable + :optional: Enables versioning on the MinIO bucket specified to :mc-cmd:`ALIAS `. @@ -60,15 +71,27 @@ Parameters Mutually exclusive with :mc-cmd:`~mc version suspend` and :mc-cmd:`~mc version info` -.. mc-cmd:: suspend +.. mc-cmd:: --exclude-folders + :optional: - Disables versioning on the MinIO bucket specified to - :mc-cmd:`ALIAS `. + Use with :mc-cmd:`mc version enable` to disable versioning on all folders (objects whose name ends with ``/``) in the specified bucket. - Mutually exclusive with :mc-cmd:`~mc version suspend` and - :mc-cmd:`~mc version info` +.. mc-cmd:: --excluded-prefixes + :optional: + + Use with :mc-cmd:`mc version enable` to disable versioning on objects matching a list of prefixes, up to 10. + The list of prefixes match all objects containing the specified strings in their prefix or name, similar to a regular expression of the form ``prefix*``. + To match objects by prefix only, use ``prefix/*``. + + For example, the following command excludes any objects containing ``_test`` or ``_temp`` in their prefix or name from versioning: + + .. code-block:: shell + :class: copyable + + mc version enable --excluded-prefixes "_test, _temp" local/my-bucket .. mc-cmd:: info + :optional: Returns the versioning configuration for the MinIO bucket specified to :mc-cmd:`ALIAS `. @@ -76,14 +99,14 @@ Parameters Mutually exclusive with :mc-cmd:`~mc version suspend` and :mc-cmd:`~mc version info` -.. mc-cmd:: ALIAS +.. mc-cmd:: suspend + :optional: - *Required* The :ref:`alias ` of a MinIO deployment and the full path - to the bucket for which to set the versioning configuration. For example: + Disables versioning on the MinIO bucket specified to + :mc-cmd:`ALIAS `. - .. code-block:: shell - - mc version enable myminio/mybucket + Mutually exclusive with :mc-cmd:`~mc version suspend` and + :mc-cmd:`~mc version info` Global Flags ~~~~~~~~~~~~ @@ -171,4 +194,4 @@ S3 Compatibility .. include:: /includes/common-minio-mc.rst :start-after: start-minio-mc-s3-compatibility - :end-before: end-minio-mc-s3-compatibility \ No newline at end of file + :end-before: end-minio-mc-s3-compatibility