diff --git a/source/extra/examples/LifecycleManagementAdmin.json b/source/extra/examples/LifecycleManagementAdmin.json new file mode 100644 index 00000000..5c24547a --- /dev/null +++ b/source/extra/examples/LifecycleManagementAdmin.json @@ -0,0 +1,24 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Action": [ + "admin:SetTier", + "admin:ListTier" + ], + "Effect": "Allow", + "Sid": "EnableRemoteTierManagement" + }, + { + "Action": [ + "s3:PutBucketLifecycle", + "s3:GetBucketLifecycle" + ], + "Resource": [ + "arn:aws:s3:::*" + ], + "Effect": "Allow", + "Sid": "EnableLifecycleManagementRules" + } + ] +} \ No newline at end of file diff --git a/source/extra/examples/LifecycleManagementUser.json b/source/extra/examples/LifecycleManagementUser.json new file mode 100644 index 00000000..4b091602 --- /dev/null +++ b/source/extra/examples/LifecycleManagementUser.json @@ -0,0 +1,27 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Action": [ + "s3:ListBucket" + ], + "Effect": "Allow", + "Resource": [ + "arn:aws:s3:::MyDestinationBucket" + ], + "Sid": "" + }, + { + "Action": [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject" + ], + "Effect": "Allow", + "Resource": [ + "arn:aws:s3:::MyDestinationBucket/*" + ], + "Sid": "" + } + ] +} \ No newline at end of file diff --git a/source/extra/examples/ReplicationAdminPolicy.json b/source/extra/examples/ReplicationAdminPolicy.json index a83edbdd..59586037 100644 --- a/source/extra/examples/ReplicationAdminPolicy.json +++ b/source/extra/examples/ReplicationAdminPolicy.json @@ -27,4 +27,4 @@ "Sid": "EnableReplicationRuleConfiguration" } ] -} \ No newline at end of file +} diff --git a/source/includes/common-minio-tiering.rst b/source/includes/common-minio-tiering.rst new file mode 100644 index 00000000..10d3f572 --- /dev/null +++ b/source/includes/common-minio-tiering.rst @@ -0,0 +1,94 @@ +.. start-create-transition-rule-desc + +Use the :mc-cmd:`mc ilm add` command to create a new transition rule +for the bucket. The following tabs contain examples for transitioning +objects on a calendar date *or* after a number of calendar days. + +.. tabs:: + + .. tab:: Transition after Calendar Date + + .. code-block:: shell + :class: copyable + + mc ilm add ALIAS/BUCKET \ + --storage-class TIERNAME \ + --transition-date DATE \ + --noncurrentversion-transition-date NONCURRENT_DAYS + + .. tab:: Transition after Calendar Days + + .. code-block:: shell + :class: copyable + + mc ilm add ALIAS/BUCKET \ + --storage-class TIERNAME \ + --transition-days DAYS \ + --noncurrentversion-transition-days NONCURRENT_DAYS + +The examples above specify the following arguments: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + :width: 100% + + * - Argument + - Description + + * - :mc-cmd:`ALIAS ` + - Specify the :mc:`alias ` of the MinIO deployment for which + you are creating the lifecycle management rule. + + * - :mc-cmd:`BUCKET ` + - Specify the full path to the bucket for which you are + creating the lifecycle management rule. + + * - :mc-cmd:`TIERNAME ` + - The remote storage tier to which MinIO transitions objects. + Specify the remote storage tier name created in the previous step. + + * - :mc-cmd:`DAYS ` + - The number of calendar days after which MinIO marks an object as + eligible for transition. + + * - :mc-cmd:`DATE ` + - The ISO-8601-formatted calendar date after which MinIO marks an object + as eligible for transition. + + * - :mc-cmd:`NONCURRENT_DAYS ` + - The number of calendar days after which MinIO marks a noncurrent + object version as eligible for transition. Omit this value to + ignore noncurrent object versions. + + This option has no effect on non-versioned buckets. + + +.. end-create-transition-rule-desc + +.. start-create-transition-user-desc + +This step creates users and policies on the MinIO deployment for supporting +lifecycle management operations. You can skip this step if the deployment +already has users with the necessary |permissions|. + +The following example uses ``Alpha`` as a placeholder :mc:`alias ` for +the MinIO deployment. Replace this value with the appropriate alias for the +MinIO deployment on which you are configuring lifecycle management rules. +Replace the password ``LongRandomSecretKey`` with a long, random, and secure +secret key as per your organizations best practices for password generation. + +.. code-block:: shell + :class: copyable + + wget -O - https://docs.min.io/minio/baremetal/examples/LifecycleManagementAdmin.json | \ + mc admin policy add Alpha LifecycleAdminPolicy /dev/stdin + mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey + mc admin policy set Alpha LifecycleAdminPolicy user=alphaLifecycleAdmin + +This example assumes that the specified +aliases have the necessary permissions for creating policies and users +on the deployment. See :ref:`minio-users` and :ref:`minio-policy` for more +complete documentation on MinIO users and policies respectively. + +.. end-create-transition-user-desc \ No newline at end of file diff --git a/source/index.rst b/source/index.rst index 0acc75b1..aef05ad8 100644 --- a/source/index.rst +++ b/source/index.rst @@ -150,6 +150,7 @@ such as versioning and replication. /introduction/minio-overview /concepts/feature-overview /tutorials/minio-installation + /lifecycle-management/lifecycle-management-overview /replication/replication-overview /security/security-overview /monitoring/monitoring-overview diff --git a/source/lifecycle-management/create-lifecycle-management-expiration-rule.rst b/source/lifecycle-management/create-lifecycle-management-expiration-rule.rst new file mode 100644 index 00000000..9ff1eee8 --- /dev/null +++ b/source/lifecycle-management/create-lifecycle-management-expiration-rule.rst @@ -0,0 +1,139 @@ +.. _minio-lifecycle-management-create-expiry-rule: + +=========================== +Automatic Object Expiration +=========================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +Each procedure on this page creates a new object lifecycle management rule that +expires objects on a MinIO bucket. This procedure supports use cases like +removing "old" objects after a certain time period or calendar date. + +.. todo: diagram + +Requirements +------------ + +Install and Configure ``mc`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This procedure uses :mc:`mc` for performing operations on the MinIO cluster. +Install :mc:`mc` on a machine with network access to both source and destination +clusters. See the ``mc`` :ref:`Installation Quickstart ` for +instructions on downloading and installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for the source MinIO cluster +and the destination S3-compatible service. Alias creation requires specifying an +access key for a user on the source and destination clusters. The specified +users must have :ref:`permissions +` for configuring +and applying expiry operations. + +.. _minio-lifecycle-management-create-expiry-rule-permissions: + +Required Permissions +~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutBucketLifecycle` +- :policy-action:`s3:GetBucketLifecycle` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster:. + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +Expire Objects after Calendar Date +---------------------------------- + +Use :mc-cmd:`mc ilm add` with :mc-cmd-option:`~mc ilm add expiry-date` to +expire bucket contents after a specific date. + +.. code-block:: shell + :class: copyable + + mc ilm add ALIAS/PATH --expiry-date "DATE" + +- Replace :mc-cmd:`ALIAS ` with the + :mc:`alias ` of the S3-compatible host. + +- Replace :mc-cmd:`PATH ` with the path to the bucket on the + S3-compatible host. + +- Replace :mc-cmd:`DATE ` with the calendar date after + which to expire the object. For example, specify "2021-01-01" to expire + objects after January 1st, 2021. + +Expire Objects after Number of Days +----------------------------------- + +Use :mc-cmd:`mc ilm add` with :mc-cmd-option:`~mc ilm add expiry-days` to +expire bucket contents a number of days after object creation: + +.. code-block:: shell + :class: copyable + + mc ilm add ALIAS/PATH --expiry-days "DAYS" + +- Replace :mc-cmd:`ALIAS ` with the + :mc:`alias ` of the S3-compatible host. + +- Replace :mc-cmd:`PATH ` with the path to the bucket on the + S3-compatible host. + +- Replace :mc-cmd:`DATE ` with the number of days after + which to expire the object. For example, specify ``30d`` to expire the + object 30 days after creation. + +Expire Versioned Objects +------------------------ + +Use :mc-cmd:`mc ilm add` to expiring noncurrent object versions and object +delete markers: + +- To expire noncurrent object versions, specify the + :mc-cmd-option:`~mc ilm add noncurrentversion-expiration-days` option. + +- To expire delete markers for objects with no remaining versions, + specify the :mc-cmd-option:`~mc ilm add expired-object-delete-marker` + option. + +.. code-block:: shell + :class: copyable + + mc ilm add ALIAS/PATH \ + --noncurrentversion-expiration-days NONCURRENT_DAYS \ + --expired-object-delete-marker EXPIRED_DAYS + +- Replace :mc-cmd:`ALIAS ` with the + :mc:`alias ` of the S3-compatible host. + +- Replace :mc-cmd:`PATH ` with the path to the bucket on the + S3-compatible host. + +- Replace :mc-cmd:`NONCURRENT_DAYS + ` with the number of days after + which to expire noncurrent object versions. For example, specify ``30d`` to + expire a version after it has been noncurrent for at least 30 days. + +- Replace :mc-cmd:`EXPIRED_DAYS + ` with the number of days after + which to expire the delete marker for an object with no remaining versions. + For example, specify ``30d`` to remove the delete marker after it has + been the only remaining "version" of that object for at least 30 days. diff --git a/source/lifecycle-management/lifecycle-management-overview.rst b/source/lifecycle-management/lifecycle-management-overview.rst new file mode 100644 index 00000000..748f17b4 --- /dev/null +++ b/source/lifecycle-management/lifecycle-management-overview.rst @@ -0,0 +1,144 @@ +.. _minio-lifecycle-management: + +=========================== +Object Lifecycle Management +=========================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 1 + +MinIO Object Lifecycle Management allows creating rules for time or date +based automatic transition or expiry of objects. For object transition, +MinIO automatically moves the object to a configured remote storage +tier. For object expiry, MinIO automatically deletes the object. + +MinIO lifecycle management is built for behavior and syntax compatibility with +:s3-docs:`AWS S3 Lifecycle Management `. For +example, you can export S3 lifecycle management rules and import them into +MinIO or vice-versa. MinIO uses JSON to describe lifecycle management rules, +and conversion to or from XML may be required. + +.. _minio-lifecycle-management-tiering: + +Object Transition ("Tiering") +----------------------------- + +MinIO supports creating object transition lifecycle management rules, where +MinIO can automatically move an object to a remote storage "tier". MinIO +supports any S3-compatible service as a remote tier *in addition to* the +following public cloud storage services: + +- :ref:`Amazon S3 ` +- :ref:`Google Cloud Storage ` +- :ref:`Microsoft Azure Blob Storage + ` + +MinIO object transition supports use cases like moving aged data from +MinIO clusters in private or public cloud infrastructure to low-cost +public cloud storage solutions. + +MinIO manages retrieving tiered objects on-the-fly without any additional +application-side logic. MinIO retains the minimum object metadata required to +support retrieving objects transitioned to a remote tier. MinIO therefore +*requires* exclusive access to the data on the remote storage tier. Object +retrieval assumes no external mutation, migration, or deletion of stored +objects. Object transition is *not* a replacement for backup/recovery +strategies such as :ref:`minio-bucket-replication`. + +Use the :mc-cmd:`mc admin tier` command to create a remote target for tiering +data to a supported Cloud Service Provider object storage You can then use the +:mc-cmd:`mc ilm add` command with one of the following commandline options to +create new transition rules on a bucket: + +- :mc-cmd-option:`mc ilm add transition-date` to transition objects after + a specified calendar date. + +- :mc-cmd-option:`mc ilm add transition-days` to transition object after + a specified number of calendar days. + +Versioned Buckets +~~~~~~~~~~~~~~~~~ + +MinIO adopts :s3-docs:`S3 behavior +` for transition rules +on :ref:`versioned buckets `. Specifically, MinIO by +default applies the transition operation to the *current* object version. + +To transition noncurrent object versions, specify the +:mc-cmd-option:`~mc ilm add noncurrentversion-transition-days` option +when creating the transition rule. + +.. _minio-lifecycle-management-expiration: + +Object Expiration +----------------- + +MinIO lifecycle management supports expiring objects on a bucket. Object +"expiration" involves performing a ``DELETE`` operation on the object. For +example, you can create a lifecycle management rule to expire any object +older than 365 days. + +.. todo: Diagram of MinIO Expiration + +Use the :mc-cmd:`mc ilm add` command with one of the following commandline +options to create new expiration rules on a bucket: + +- :mc-cmd-option:`mc ilm add expiry-date` to expire objects after + a specified calendar date. + +- :mc-cmd-option:`mc ilm add expiry-days` to expire objects after a + specified number of calendar days. + +Versioned Buckets +~~~~~~~~~~~~~~~~~ + +MinIO adopts :s3-docs:`S3 behavior +` for expiration rules +on :ref:`versioned buckets `. MinIO has two +specific default behaviors for versioned buckets: + +- MinIO applies the expiration option to only the *current* object version by + creating a ``DeleteMarker`` as is normal with versioned delete. + + To expire noncurrent object versions, specify the + :mc-cmd-option:`~mc ilm add noncurrentversion-expiration-days` option + when creating the expiration rule. + +- MinIO does not expire ``DeleteMarkers`` *even if* no other versions of + that object exist. + + To expire delete markers when there are no remaining versions for that + object, specify the :mc-cmd-option:`~mc ilm add expired-object-delete-marker` + option when creating the expiration rule. + +.. _minio-lifecycle-management-scanner: + +Lifecycle Management Object Scanner +----------------------------------- + +MinIO uses a built-in scanner to actively check objects against all +configured lifecycle management rules. The scanner is a low-priority process +that yields to high IO workloads to prevent performance spikes triggered +by rule timing. The scanner may therefore not detect an object as eligible +for a configured transition or expiration lifecycle rule until *after* +the lifecycle rule period has passed. + +Delayed application of lifecycle management rules is typically associated to +limited node resources and cluster size. Scanner speed tends to slow as +clusters grow as more time is required to visit all buckets and objects. +This can be exacerbated if the cluster hardware is undersized for regular +workloads, as the scanner will yield to high cluster load to avoid performance +loss. Consider regularly checking cluster metrics, capacity, and resource +usage to ensure the cluster hardware is scaling alongside cluster and workload +growth. + +.. toctree:: + :hidden: + + /lifecycle-management/transition-objects-to-s3.rst + /lifecycle-management/transition-objects-to-gcs.rst + /lifecycle-management/transition-objects-to-azure.rst + /lifecycle-management/create-lifecycle-management-expiration-rule.rst diff --git a/source/lifecycle-management/transition-objects-to-azure.rst b/source/lifecycle-management/transition-objects-to-azure.rst new file mode 100644 index 00000000..3e05e53a --- /dev/null +++ b/source/lifecycle-management/transition-objects-to-azure.rst @@ -0,0 +1,197 @@ +.. _minio-lifecycle-management-transition-to-azure: + +====================================== +Transition Objects from MinIO to Azure +====================================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +The procedure on this page creates a new object lifecycle management rule that +transition objects from a MinIO bucket to a remote storage tier on the +:abbr:`Azure (Microsoft Azure)` storage backend. This procedure supports use +cases like moving aged data to low-cost public cloud storage solutions after a +certain time period or calendar date. + +.. todo: diagram + +Requirements +------------ + +Install and Configure ``mc`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This procedure uses :mc:`mc` for performing operations on the MinIO cluster. +Install :mc:`mc` on a machine with network access to both source and destination +clusters. See the ``mc`` :ref:`Installation Quickstart ` for +instructions on downloading and installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for the source MinIO cluster. +Alias creation requires specifying an access key for a user on the source and +destination clusters. The specified users must have :ref:`permissions +` for configuring +and applying transition operations. + +.. _minio-lifecycle-management-transition-to-azure-permissions: + +Required MinIO Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutBucketLifecycle` +- :policy-action:`s3:GetBucketLifecycle` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster:. + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +.. _minio-lifecycle-management-transition-to-azure-permissions-remote: + +Required Azure Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Object transition lifecycle management rules require additional permissions on +the remote storage tier. Specifically, MinIO requires the +:abbr:`Azure (Microsoft Azure)` credentials provide read, write, list, and +delete permissions for the remote bucket. + +Refer to the `Azure RBAC +`__ +documentation for more complete guidance on configuring the required +permissions. + +Considerations +-------------- + +Lifecycle Management Object Scanner +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses a scanner process to check objects against all configured +lifecycle management rules. Slow scanning due to high IO workloads or +limited system resources may delay application of lifecycle management +rules. See :ref:`minio-lifecycle-management-scanner` for more information. + +Exclusive Bucket Access +~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO retains the minimum object metadata required to +support retrieving objects transitioned to a remote tier. MinIO therefore +*requires* exclusive access to the data on the remote storage tier. Object +retrieval assumes no external mutation, migration, or deletion of stored +objects. + +MinIO also ignores any objects in the remote bucket or bucket prefix not +explicitly managed by MinIO. + +Procedure +--------- + +1) Configure User Accounts and Policies for Lifecycle Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. |permissions| replace:: :ref:`permissions ` + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-user-desc + :end-before: end-create-transition-user-desc + +2) Configure the Remote Storage Tier +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc admin tier add` command to add a new remote storage tier: + +.. code-block:: shell + :class: copyable + + mc admin tier add azure TARGET TIER_NAME \ + --endpoint https://HOSTNAME + --bucket BUCKET \ + --prefix PREFIX + --account-name ACCOUNT \ + --account-key KEY \ + --region REGION + + +The example above uses the following arguments: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + :width: 100% + + * - Argument + - Description + + * - :mc-cmd:`TARGET ` + - The :mc:`alias ` of the MinIO deployment on which to configure + the remote tier. + + * - :mc-cmd:`TIER_NAME ` + - The name to associate with the new :abbr:`Azure (Microsoft Azure)` blob + remote storage tier. This value is required in the next step. + + * - :mc-cmd:`HOSTNAME ` + - The URL endpoint for the :abbr:`Azure (Microsoft Azure)` storage + backend. + + * - :mc-cmd:`BUCKET ` + - The name of the bucket on the :abbr:`Azure (Microsoft Azure)` storage + backend to which MinIO transitions objects. + + * - :mc-cmd:`PREFIX ` + - The optional bucket prefix within which MinIO transitions objects. + Omit this argument to transition objects to the bucket root. + + * - :mc-cmd:`ACCOUNT ` + - The account name MinIO uses to access the bucket. The account name + *must* correspond to an :abbr:`Azure (Microsoft Azure)` user with the + required :ref:`permissions + `. + + * - :mc-cmd:`KEY ` + - The corresponding key for the specified ``ACCOUNT``. + + * - :mc-cmd:`REGION ` + - The :abbr:`Azure (Microsoft Azure)` blob storage region of the specified + ``BUCKET``. You can safely omit this option if the ``HOSTNAME`` includes + the region. + +3) Create and Apply the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-rule-desc + :end-before: end-create-transition-rule-desc + + +3) Verify the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc ilm list` command to review the configured transition +rules: + +.. code-block:: shell + :class: copyable + + mc ilm list ALIAS/PATH --transition + +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` + of the MinIO deployment. + +- Replace :mc-cmd:`PATH ` with the name of the bucket for + which to retrieve the configured lifecycle management rules. diff --git a/source/lifecycle-management/transition-objects-to-gcs.rst b/source/lifecycle-management/transition-objects-to-gcs.rst new file mode 100644 index 00000000..3c9db83c --- /dev/null +++ b/source/lifecycle-management/transition-objects-to-gcs.rst @@ -0,0 +1,195 @@ +.. _minio-lifecycle-management-transition-to-gcs: + +==================================== +Transition Objects from MinIO to GCS +==================================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +The procedure on this page creates a new object lifecycle management rule that +transition objects from a MinIO bucket to a remote storage tier on the Google +Cloud Storage backend. This procedure supports use cases like moving aged data +to low-cost public cloud storage solutions after a certain time period or +calendar date. + +.. todo: diagram + +Requirements +------------ + +Install and Configure ``mc`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This procedure uses :mc:`mc` for performing operations on the MinIO cluster. +Install :mc:`mc` on a machine with network access to both source and destination +clusters. See the ``mc`` :ref:`Installation Quickstart ` for +instructions on downloading and installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for the source MinIO cluster. +Alias creation requires specifying an access key for a user on the source and +destination clusters. The specified users must have :ref:`permissions +` for configuring and +applying transition operations. + +.. _minio-lifecycle-management-transition-to-gcs-permissions: + +Required MinIO Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutBucketLifecycle` +- :policy-action:`s3:GetBucketLifecycle` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster:. + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +.. _minio-lifecycle-management-transition-to-gcs-permissions-remote: + +Required GCS Permissions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Object transition lifecycle management rules require additional permissions on +the remote storage tier. Specifically, MinIO requires the +:abbr:`GCS (Google Cloud Storage)` credentials provide read, write, list, and +delete permissions for the remote bucket. + +Refer to the `GCS IAM permissions +`__ +documentation for more complete guidance on configuring the required +permissions. + +Considerations +-------------- + +Lifecycle Management Object Scanner +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses a scanner process to check objects against all configured +lifecycle management rules. Slow scanning due to high IO workloads or +limited system resources may delay application of lifecycle management +rules. See :ref:`minio-lifecycle-management-scanner` for more information. + +Exclusive Bucket Access +~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO retains the minimum object metadata required to +support retrieving objects transitioned to a remote tier. MinIO therefore +*requires* exclusive access to the data on the remote storage tier. Object +retrieval assumes no external mutation, migration, or deletion of stored +objects. + +MinIO also ignores any objects in the remote bucket or bucket prefix not +explicitly managed by MinIO. + +Procedure +--------- + +1) Configure User Accounts and Policies for Lifecycle Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. |permissions| replace:: :ref:`permissions ` + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-user-desc + :end-before: end-create-transition-user-desc + +2) Configure the Remote Storage Tier +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc admin tier add` command to add a new Google Cloud Storage +service as the remote storage tier: + +.. code-block:: shell + :class: copyable + + mc admin tier add gcs TARGET TIER_NAME \ + --endpoint https://HOSTNAME \ + --bucket BUCKET \ + --prefix PREFIX + --credentials-file CREDENTIALS \ + --region REGION + +The example above uses the following arguments: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + :width: 100% + + * - Argument + - Description + + * - :mc-cmd:`TARGET ` + - The :mc:`alias ` of the MinIO deployment on which to configure + the :abbr:`GCS (Google Cloud Storage)` remote tier. + + * - :mc-cmd:`TIER_NAME ` + - The name to associate with the new :abbr:`GCS (Google Cloud Storage)` + remote storage tier. This value is required in the next step. + + * - :mc-cmd:`HOSTNAME ` + - The URL endpoint for the :abbr:`GCS (Google Cloud Storage)` storage + backend. + + * - :mc-cmd:`BUCKET ` + - The name of the bucket on the :abbr:`GCS (Google Cloud Storage)` storage + backend to which MinIO transitions objects. + + * - :mc-cmd:`PREFIX ` + - The optional bucket prefix within which MinIO transitions objects. + Omit this argument to transition objects to the bucket root. + + * - :mc-cmd:`CREDENTIALS ` + - The `credential file + `__ for a + user on the remote GCS tier. The specified user credentials *must* + correspond to a GCS user with the required + :ref:`permissions + `. + + * - :mc-cmd:`REGION ` + - The :abbr:`GCS (Google Cloud Storage)` region of the specified + ``BUCKET``. You can safely omit this + option if the ``HOSTNAME`` includes the region. + +3) Create and Apply the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-rule-desc + :end-before: end-create-transition-rule-desc + + +3) Verify the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc ilm list` command to review the configured transition +rules: + +.. code-block:: shell + :class: copyable + + mc ilm list ALIAS/PATH --transition + +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` + of the MinIO deployment. + +- Replace :mc-cmd:`PATH ` with the name of the bucket for + which to retrieve the configured lifecycle management rules. diff --git a/source/lifecycle-management/transition-objects-to-s3.rst b/source/lifecycle-management/transition-objects-to-s3.rst new file mode 100644 index 00000000..c8187d8b --- /dev/null +++ b/source/lifecycle-management/transition-objects-to-s3.rst @@ -0,0 +1,213 @@ +.. _minio-lifecycle-management-transition-to-s3: + +=================================== +Transition Objects from MinIO to S3 +=================================== + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +The procedure on this page creates a new object lifecycle management rule that +transition objects from a MinIO bucket to a remote storage tier on the Amazon +Web Services S3 storage backend *or* an S3-compatible service. This procedure +supports use cases such as tiering objects to low-cost or archival storage after +a certain time period or calendar date. + +.. todo: diagram + +Requirements +------------ + +Install and Configure ``mc`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This procedure uses :mc:`mc` for performing operations on the MinIO cluster. +Install :mc:`mc` on a machine with network access to both source and destination +clusters. See the ``mc`` :ref:`Installation Quickstart ` for +instructions on downloading and installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for the source MinIO cluster. +Alias creation requires specifying an access key for a user on the source and +destination clusters. The specified users must have :ref:`permissions +` for configuring and +applying transition operations. + +.. _minio-lifecycle-management-transition-to-s3-permissions: + +Required MinIO Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutBucketLifecycle` +- :policy-action:`s3:GetBucketLifecycle` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster:. + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +.. _minio-lifecycle-management-transition-to-s3-permissions-remote: + +Required S3 Permissions +~~~~~~~~~~~~~~~~~~~~~~~ + +Object transition lifecycle management rules require additional permissions +on the remote storage tier. Specifically, MinIO requires the remote +tier credentials provide read, write, list, and delete permissions for the +remote bucket. + +For example, the following policy provides the necessary permission +for transitioning objects into and out of the remote tier: + +.. literalinclude:: /extra/examples/LifecycleManagementUser.json + :language: json + :class: copyable + +Modify the ``Resource`` for the bucket into which MinIO tiers objects. + +Refer to the :aws-docs:`Amazon S3 Permissions +` +documentation for more complete guidance on configuring the required +permissions. + +Considerations +-------------- + +Lifecycle Management Object Scanner +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses a scanner process to check objects against all configured +lifecycle management rules. Slow scanning due to high IO workloads or +limited system resources may delay application of lifecycle management +rules. See :ref:`minio-lifecycle-management-scanner` for more information. + +Exclusive Bucket Access +~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO retains the minimum object metadata required to +support retrieving objects transitioned to a remote tier. MinIO therefore +*requires* exclusive access to the data on the remote storage tier. Object +retrieval assumes no external mutation, migration, or deletion of stored +objects. + +MinIO also ignores any objects in the remote bucket or bucket prefix not +explicitly managed by MinIO. + +Procedure +--------- + +1) Configure User Accounts and Policies for Lifecycle Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. |permissions| replace:: :ref:`permissions ` + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-user-desc + :end-before: end-create-transition-user-desc + +2) Configure the Remote Storage Tier +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc admin tier add` command to add an Amazon S3 service as the +new remote storage tier: + +.. code-block:: shell + :class: copyable + + mc admin tier add s3 TARGET TIER_NAME \ + --endpoint https://HOSTNAME \ + --bucket BUCKET \ + --prefix PREFIX + --access-key ACCESS_KEY \ + --secret-key SECRET_KEY \ + --region REGION \ + --storage-class STORAGE_CLASS + +The example above uses the following arguments: + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + :width: 100% + + * - Argument + - Description + + * - :mc-cmd:`TARGET ` + - The :mc:`alias ` of the MinIO deployment on which to configure + the S3 remote tier. + + * - :mc-cmd:`TIER_NAME ` + - The name to associate with the new S3 remote storage tier. This value + is required in the next step. + + * - :mc-cmd:`HOSTNAME ` + - The URL endpoint for the S3 storage backend. + + * - :mc-cmd:`BUCKET ` + - The name of the bucket on the S3 storage backend to which MinIO + transitions objects. + + * - :mc-cmd:`PREFIX ` + - The optional bucket prefix within which MinIO transitions objects. + Omit this argument to transition objects to the bucket root. + + * - :mc-cmd:`ACCESS_KEY ` + - The S3 access key MinIO uses to access the bucket. The + access key *must* correspond to an IAM user with the + required + :ref:`permissions + `. + + * - :mc-cmd:`SECRET_KEY ` + - The corresponding secret key for the specified ``ACCESS_KEY``. + + * - :mc-cmd:`REGION ` + - The AWS S3 region of the specified ``BUCKET``. You can safely omit this + option if the ``HOSTNAME`` includes the region. + + * - :mc-cmd:`STORAGE_CLASS ` + - The S3 storage class to which MinIO transitions objects. Specify + one of the following supported storage classes: + + - ``STANDARD`` + - ``REDUCED`` + +3) Create and Apply the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-rule-desc + :end-before: end-create-transition-rule-desc + + +3) Verify the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc-cmd:`mc ilm list` command to review the configured transition +rules: + +.. code-block:: shell + :class: copyable + + mc ilm list ALIAS/PATH --transition + +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` + of the MinIO deployment. + +- Replace :mc-cmd:`PATH ` with the name of the bucket for + which to retrieve the configured lifecycle management rules. diff --git a/source/reference/minio-cli/minio-mc-admin.rst b/source/reference/minio-cli/minio-mc-admin.rst index 1d438f6b..3fd8b2e1 100644 --- a/source/reference/minio-cli/minio-mc-admin.rst +++ b/source/reference/minio-cli/minio-mc-admin.rst @@ -90,6 +90,11 @@ The following table lists :mc-cmd:`mc admin` commands: :start-after: start-mc-admin-service-desc :end-before: end-mc-admin-service-desc + * - :mc:`mc admin tier` + - .. include:: /reference/minio-cli/minio-mc-admin/mc-admin-tier.rst + :start-after: start-mc-admin-tier-desc + :end-before: end-mc-admin-tier-desc + * - :mc:`mc admin top` - .. include:: /reference/minio-cli/minio-mc-admin/mc-admin-top.rst :start-after: start-mc-admin-top-desc diff --git a/source/reference/minio-cli/minio-mc-admin/mc-admin-tier.rst b/source/reference/minio-cli/minio-mc-admin/mc-admin-tier.rst new file mode 100644 index 00000000..7020d02a --- /dev/null +++ b/source/reference/minio-cli/minio-mc-admin/mc-admin-tier.rst @@ -0,0 +1,386 @@ +================= +``mc admin tier`` +================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +.. mc:: mc admin tier + +Description +----------- + +.. start-mc-admin-tier-desc + +The :mc:`mc admin tier` command configures a remote supported S3-compatible +service for supporting MinIO +:ref:`Lifecycle Management: Object Transition ("Tiering") +`. + +.. end-mc-admin-tier-desc + +.. admonition:: Use ``mc admin`` on MinIO Deployments Only + :class: note + + .. include:: /includes/facts-mc-admin.rst + :start-after: start-minio-only + :end-before: end-minio-only + +Supported S3 Services +~~~~~~~~~~~~~~~~~~~~~ + +:mc:`mc admin tier` supports *only* the following S3-compatible services +as a remote target for object tiering: + +- Amazon S3 +- Google Cloud Storage +- Azure Blob Storage + +Required Permissions +~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutBucketLifecycle` +- :policy-action:`s3:GetBucketLifecycle` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster:. + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +Transition Permissions +++++++++++++++++++++++ + +Object transition lifecycle management rules require additional permissions +on the remote storage tier. Specifically, MinIO requires the remote +tier credentials provide read, write, list, and delete permissions. + +For example, if the remote storage tier implements AWS IAM policy-based +access control, the following policy provides the necessary permission +for transitioning objects into and out of the remote tier: + +.. literalinclude:: /extra/examples/LifecycleManagementUser.json + :language: json + :class: copyable + +Modify the ``Resource`` for the bucket into which MinIO tiers objects. + +Defer to the documentation for the supported tiering targets for more complete +information on configuring users and permissions to support MinIO tiering: + +- :aws-docs:`Amazon S3 Permissions ` +- `Google Cloud Storage Access Control `__ +- `Authorizing access to data in Azure storage `__ + + +Syntax +------ + +.. mc-cmd:: add + :fullpath: + + Creates a new remote storage tier for transitioning objects using MinIO + lifecycle management rules. + + .. important:: + + MinIO does not support removing remote storage tiers. Ensure the + storage backend supports the intended workload *prior* to + adding it as a remote tier target. + + The command has the following syntax: + + .. code-block:: shell + :class: copyable + + mc admin tier add TIER_TYPE TARGET TIER_NAME [FLAGS] + + The command accepts the following arguments: + + .. mc-cmd:: TIER_TYPE + + *Required* + + The Cloud Service Provider storage backend ("Tier") to which MinIO + transitions objects. Specify *one* of the following supported values: + + .. list-table:: + :stub-columns: 1 + :width: 100% + :widths: 30 70 + + * - ``s3`` + - Use AWS S3 *or* a remote MinIO deployment as the storage + backend for the new Tier. + + Requires specifying the following additional options: + + - :mc-cmd-option:`~mc admin tier add access-key` + - :mc-cmd-option:`~mc admin tier add secret-key` + + * - ``azure`` + - Use :abbr:`Azure (Microsoft Azure)` Blob Storage as the storage + backend for the new Tier. + + Requires specifying the following additional options: + + - :mc-cmd-option:`~mc admin tier add account-name` + - :mc-cmd-option:`~mc admin tier add account-key` + + * - ``gcs`` + - Use :abbr:`GCP (Google Cloud Platform)` Cloud Storage as the + storage backend for the new Tier. + + Requires specifying the following additional option: + + - :mc-cmd-option:`~mc admin tier add credentials-file` + + .. mc-cmd:: TARGET + + *Required* + + The :mc-cmd:`alias ` of a configured MinIO deployment on which + the command creates the new remote tier. + + .. mc-cmd:: TIER_NAME + + *Required* + + The name to associate with the new remote tier. The name *must* + be unique across all configured tiers on the MinIO cluster. + + .. mc-cmd:: endpoint + :option: + + *Required* + + The URL endpoint for the cloud service provider. The URL endpoint + *must* resolve to the provider specified to + :mc-cmd:`~mc admin tier add TIER_TYPE`. + + .. mc-cmd:: access-key + :option: + + *Required* + + The access key for a user on the remote S3 tier. The user must + have permission to perform read/write/list/delete operations on the + remote bucket or bucket prefix. + + Required if :mc-cmd:`~mc admin tier add TIER_TYPE` is ``s3``. + This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: secret-key + :option: + + *Required* + + The secret key for a user on the remote S3 tier. + + Required if :mc-cmd:`~mc admin tier add TIER_TYPE` is ``s3``. + This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: account-name + :option: + + *Required* + + The account name for a user on the remote Azure tier. The user + must have permission to perform read/write/list/delete operations on the + remote bucket or bucket prefix. + + Required if :mc-cmd:`~mc admin tier add TIER_TYPE` is ``azure``. + This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: account-key + :option: + + *Required* + + The account key for a user on the remote Azure tier. + + Required if :mc-cmd:`~mc admin tier add TIER_TYPE` is ``azure``. + This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: credentials-file + :option: + + *Required* + + The `credential file + `__ for a + user on the remote GCS tier. The user must have permission to perform + read/write/list/delete operations on the remote bucket or bucket prefix. + + Required if :mc-cmd:`~mc admin tier add TIER_TYPE` is ``gcs``. + This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: bucket + :option: + + *Required* + + The bucket on the remote tier to which MinIO transitions objects. + + .. mc-cmd:: prefix + :option: + + *Optional* + + The prefix path for the specified :mc-cmd:`~mc admin tier add bucket` + to which MinIO transitions objects. + + Omit this field to transition objects into the bucket root. + + .. mc-cmd:: storage-class + :option: + + *Optional* + + The AWS storage class to use for objects transitioned by + MinIO. MinIO supports only the following storage classes: + + - ``STANDARD`` + - ``REDUCED_REDUNDANCY`` + + Defaults to ``S3_STANDARD`` if omitted. + + This option only applies if :mc-cmd:`~mc admin tier add TIER_TYPE` is + ``s3``. This option has no effect for any other value of ``TIER_TYPE``. + + .. mc-cmd:: region + :option: + + *Optional* + + The S3 backend region for the specified + :mc-cmd:`~mc admin tier add TIER_TYPE`, such as ``us-west-1``. + + This option only applies if :mc-cmd:`~mc admin tier add TIER_TYPE` is + ``s3``. This option has no effect for any other value of ``TIER_TYPE``. + +.. mc-cmd:: edit + :fullpath: + + Modify or remove a remote storage tier from a MinIO cluster. Remote + storage tiers support transitioning objects using MinIO lifecycle + management rules. + + The command has the following syntax: + + .. code-block:: shell + :class: copyable + + mc admin tier edit TARGET TIER_NAME [FLAGS] + + The command accepts the following arguments: + + .. mc-cmd:: TARGET + + *Required* + + The :mc-cmd:`alias ` of a configured MinIO deployment. + + .. mc-cmd:: TIER_NAME + + *Required* + + The name of the remote tier the command modifies. The value + corresponds to the :mc-cmd:`mc admin tier add TIER_NAME` specified + when creating the remote tier. + + .. mc-cmd:: access-key + :option: + + *Optional* + + The access key for a user on the remote S3 tier. The user must + have permission to perform read/write/list/delete operations on the + remote bucket or bucket prefix. + + This option only applies to remote storage tiers with + :mc-cmd:`~mc admin tier add TIER_TYPE` is ``s3``. + This option has no effect for any other ``TIER_TYPE``. + + .. mc-cmd:: secret-key + :option: + + *Optional* + + The secret key for a user on the remote S3 tier. + + This option only applies to remote storage tiers with + :mc-cmd:`~mc admin tier add TIER_TYPE` is ``s3``. + This option has no effect for any other ``TIER_TYPE``. + + .. mc-cmd:: account-name + :option: + + *Required* + + The account name for a user on the remote Azure tier. The user + must have permission to perform read/write/list/delete operations on the + remote bucket or bucket prefix. + + This option only applies to remote storage tiers with + :mc-cmd:`~mc admin tier add TIER_TYPE` is ``azure``. + This option has no effect for any other ``TIER_TYPE``. + + .. mc-cmd:: account-key + :option: + + *Required* + + The account key for a user on the remote Azure tier. + + This option only applies to remote storage tiers with + :mc-cmd:`~mc admin tier add TIER_TYPE` is ``azure``. + This option has no effect for any other ``TIER_TYPE``. + + .. mc-cmd:: credentials-file + :option: + + *Required* + + The credential file for a user on the remote GCS tier. The user must have + permission to perform read/write/list/delete operations on the remote bucket + or bucket prefix. + + This option only applies to remote storage tiers with + :mc-cmd:`~mc admin tier add TIER_TYPE` is ``gcs``. + This option has no effect for any other ``TIER_TYPE``. + +.. mc-cmd:: ls + :fullpath: + + List all remote storage tiers on a MinIO cluster. Remote storage tiers + support transitioning objects using MinIO lifecycle management rules. + + The command has the following syntax: + + .. code-block:: shell + :class: copyable + + mc admin tier ls TARGET [FLAGS] + + The command accepts the following arguments: + + .. mc-cmd:: TARGET + + *Required* + + The :mc-cmd:`alias ` of a configured MinIO deployment. diff --git a/source/reference/minio-cli/minio-mc/mc-ilm.rst b/source/reference/minio-cli/minio-mc/mc-ilm.rst index 6b4ebc36..9af024cf 100644 --- a/source/reference/minio-cli/minio-mc/mc-ilm.rst +++ b/source/reference/minio-cli/minio-mc/mc-ilm.rst @@ -22,6 +22,24 @@ information. .. end-mc-ilm-desc +Lifecycle Management Object Scanner +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses a scanner process to check objects against all configured +lifecycle management rules. Slow scanning due to high IO workloads or +limited system resources may delay application of lifecycle management +rules. See :ref:`minio-lifecycle-management-scanner` for more information. + +Expiry vs Transition +~~~~~~~~~~~~~~~~~~~~ + +MinIO supports specifying both expiry and transition rules in the same +bucket or bucket prefix. MinIO can execute an expiration rule on an object +regardless of its transition status. Use +:mc-cmd:`mc ilm list` to review the currently configured object lifecycle +management rules for any potential interactions between expiry and transition +rules. + Examples -------- @@ -34,14 +52,7 @@ expire bucket contents after a specific date. .. code-block:: shell :class: copyable - mc ilm add --id "RULE" --expiry-date "DATE" ALIAS/PATH - -- Replace :mc-cmd:`RULE ` with the unique name of the lifecycle - management rule. - -- Replace :mc-cmd:`DATE ` with the future date after - which to expire the object. For example, specify "2021-01-01" to expire - objects after January 1st, 2021. + mc ilm add ALIAS/PATH --expiry-date "DATE" - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. @@ -49,6 +60,10 @@ expire bucket contents after a specific date. - Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. +- Replace :mc-cmd:`DATE ` with the calendar date after + which to expire the object. For example, specify "2021-01-01" to expire + objects after January 1st, 2021. + Expire Bucket Contents After Number of Days ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -58,14 +73,7 @@ expire bucket contents a number of days after object creation: .. code-block:: shell :class: copyable - mc ilm add --id "RULE" --expiry-days "DAYS" ALIAS/PATH - -- Replace :mc-cmd:`RULE ` with the unique name of the lifecycle - management rule. - -- Replace :mc-cmd:`DATE ` with the number of days after - which to expire the object. For example, specify ``30d`` to expire the - object 30 days after creation. + mc ilm add ALIAS/PATH --expiry-days "DAYS" - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. @@ -73,6 +81,10 @@ expire bucket contents a number of days after object creation: - Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. +- Replace :mc-cmd:`DATE ` with the number of days after + which to expire the object. For example, specify ``30d`` to expire the + object 30 days after creation. + List Bucket Lifecycle Management Rules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -83,10 +95,10 @@ Use :mc-cmd:`mc ilm list` to list a bucket's lifecycle management rules: mc ilm list ALIAS/PATH -- Replace :mc-cmd:`ALIAS ` with the +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. -- Replace :mc-cmd:`PATH ` with the path to the bucket on the +- Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. Remove a Bucket Lifecycle Management Rule @@ -99,13 +111,13 @@ Use :mc-cmd:`mc ilm remove` to remove a bucket lifecycle management rule: mc ilm remove --id "RULE" ALIAS/PATH -- Replace :mc-cmd:`RULE ` with the unique name of the lifecycle +- Replace :mc-cmd:`RULE ` with the unique name of the lifecycle management rule. -- Replace :mc-cmd:`ALIAS ` with the +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. -- Replace :mc-cmd:`PATH ` with the path to the bucket on the +- Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. @@ -121,7 +133,7 @@ Syntax .. code-block:: shell :class: copyable - mc ilm list [FLAGS] TARGET + mc ilm list TARGET [FLAGS] The subcommand supports the following arguments: @@ -161,36 +173,27 @@ Syntax .. mc-cmd:: add :fullpath: - Adds or modifies bucket lifecycle management rules. The command has + Adds a new bucket lifecycle management rules. The command has the following syntax: .. code-block:: shell :class: copyable - mc ilm add [FLAGS] TARGET + mc ilm add TARGET [FLAGS] .. mc-cmd:: TARGET *Required* - The full path to the bucket from which to add or modify the lifecycle + The full path to the bucket from which to add the lifecycle management rule. Specify the :mc-cmd:`alias ` of a configured S3 service as the prefix to the ``TARGET`` path. - Specify all ``[FLAGS]`` *prior* to the ``TARGET``. - For example: .. code-block:: shell - mc ilm list [FLAGS] play/mybucket - - .. mc-cmd:: id - :option: - - *Required* The unique name of the rule. Specify the - :mc-cmd-option:`mc ilm add id` of an existing rule to modify the - lifecycle configuration of that rule. + mc ilm add play/mybucket [FLAGS] .. mc-cmd:: prefix :option: @@ -208,46 +211,190 @@ Syntax One or more ampersand ``&``-delimited key-value pairs describing the object tags to which to apply the lifecycle configuration rule. + This option is mutually exclusive with the following option: + + - :mc-cmd-option:`~mc ilm add expired-object-delete-marker` + .. mc-cmd:: expiry-date :option: - The ISO-8601-formatted date after which MinIO removes objects - covered by the rule. Specifying a date that is *prior* to the - current date marks all objects covered by the rule for removal. - + The ISO-8601-formatted calendar date until which MinIO retains an object + after being created. MinIO marks the object for deletion once the + system host datetime passes that calendar date. + + Specifying a calendar date that is *prior* to the current system host + datetime marks all objects covered by the rule for deletion. + + For versioned buckets, the expiry rule applies only to the *current* + object version. Use the + :mc-cmd-option:`~mc ilm add noncurrentversion-expiration-days` option + to apply expiration behavior to noncurrent object versions. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. + .. mc-cmd:: expiry-days :option: - The number of days from object creation after which MinIO removes - objects covered by the rule. + The number of days to retain an object after being created. MinIO + marks the object for deletion after the specified number of days pass. + + For versioned buckets, the expiry rule applies only to the *current* + object version. Use the + :mc-cmd-option:`~mc ilm add noncurrentversion-expiration-days` option + to apply expiration behavior to noncurrent object versions. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. + + .. mc-cmd:: noncurrentversion-expiration-days + :option: + + The number of days to retain an object version after becoming + *non-current* (i.e. a different version of that object is now the `HEAD`). + MinIO marks noncurrent object versions for deletion after the + specified number of days pass. + + This option has the same behavior as the + S3 ``NoncurrentVersionExpiration`` action. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. + + .. mc-cmd:: expired-object-delete-marker + :option: + + Specify ``true`` to direct MinIO to remove delete markers for + objects with no remaining object versions. Specifically, the delete + marker is the *only* remaining "version" of the given object. + + Defaults to ``false``. + + This option is mutually exclusive with the following option: + + - :mc-cmd-option:`~mc ilm add tags` + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. .. mc-cmd:: transition-date :option: - The ISO-8601-formatted date after which MinIO transitions objects - covered by the rule to the specified ``--storage-class``. - Specifying a date that is *prior* to the current date marks all - objects covered by the rule for transition. + The ISO-8601-formatted calendar date after which MinIO marks an object as + eligible for transition to the remote tier. MinIO transitions the object + to the configured remote storage tier specified to the + :mc-cmd-option:`~mc ilm add storage-class` once the system host datetime + passes that calendar date. + + For versioned buckets, the transition rule applies only to the *current* + object version. Use the + :mc-cmd-option:`~mc ilm add noncurrentversion-transition-days` option + to apply transition behavior to noncurrent object versions. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. .. mc-cmd:: transition-days :option: - The number of days from object creation after which MinIO - transitions objects covered by the rule to the specified - ``--storage-class``. + The number of calendar days from object creation after which MinIO marks + an object as eligible for transition. MinIO transitions the object to the + configured remote storage tier specified to the + :mc-cmd-option:`~mc ilm add storage-class`. + + For versioned buckets, the transition rule applies only to the *current* + object version. Use the + :mc-cmd-option:`~mc ilm add noncurrentversion-transition-days` option + to apply transition behavior to noncurrent object versions. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. + + .. mc-cmd:: noncurrentversion-transition-days + :option: + + The number of days an object has been non-current (i.e. replaced by a + newer version of that same object) after which MinIO marks the object + version as eligible for transition. MinIO transitions the object to the + configured remote storage tier specified to the + :mc-cmd-option:`~mc ilm add storage-class` once the system host datetime + passes that calendar date. + + This option has no effect on non-versioned buckets. + + This option has the same behavior as the + S3 ``NoncurrentVersionTransition`` action. + + MinIO uses a scanner process to check objects against all configured + lifecycle management rules. Slow scanning due to high IO workloads or + limited system resources may delay application of lifecycle management + rules. See :ref:`minio-lifecycle-management-scanner` for more information. .. mc-cmd:: storage-class :option: - The Amazon S3 storage class to transition objects covered by the - rule. See :s3-docs:`Transition objects using Amazon S3 Lifecycle - ` for more - information on S3 storage classes. + The remote storage tier to which MinIO + :ref:`transition objects `. + Specify a remote storage tier created by :mc-cmd:`mc admin tier`. + + If using :mc-cmd:`mc ilm add` against an Amazon S3 service, this argument + is the Amazon S3 storage class to transition objects covered by the rule. + See :s3-docs:`Transition objects using Amazon S3 Lifecycle + ` for more information + on S3 storage classes. .. mc-cmd:: disable :option: - Disables the rule with matching :mc-cmd-option:`~mc ilm add id`. + Disables the rule. + +.. mc-cmd:: edit + :fullpath: + + Edits an existing lifecycle management rule in the bucket. The command + has the following syntax: + + .. code-block:: shell + :class: copyable + + mc ilm edit --id "RULE_ID" TARGET [FLAGS] + + The command supports the following arguments: + + .. mc-cmd:: TARGET + + *Required* + + The full path to the bucket from which to modify the + specified lifecycle management rule. Specify the :mc-cmd:`alias + ` of a configured S3 service as the prefix to the + ``TARGET`` path. + + For example: + + .. code-block:: shell + + mc ilm edit --id "RULE_ID" play/mybucket [FLAGS] + + .. mc-cmd:: id + :option: + + *Required* + + The unique ID of the rule. Use :mc-cmd:`mc ilm list` to list bucket rules + and retrieve the ``id`` for the rule you want to modify. .. mc-cmd:: remove :fullpath: @@ -277,7 +424,10 @@ Syntax .. mc-cmd:: id - *Required* The unique name of the rule. + *Required* + + The unique ID of the rule. Use :mc-cmd:`mc ilm list` to list bucket rules + and retrieve the ``id`` for the rule you want to remove. Mutually exclusive with :mc-cmd-option:`mc ilm remove all` diff --git a/source/security/IAM/iam-policies.rst b/source/security/IAM/iam-policies.rst index 16561c29..7b891569 100644 --- a/source/security/IAM/iam-policies.rst +++ b/source/security/IAM/iam-policies.rst @@ -547,6 +547,16 @@ services: Allows getting bucket targets +.. policy-action:: admin:SetTier + + Allows creating and modifying remote storage tiers using the + :mc-cmd:`mc admin tier` command. + +.. policy-action:: admin:ListTier + + Allows listing configured remote storage tiers using the + :mc-cmd:`mc admin tier` command. + .. _minio-policy-conditions: Supported S3 Policy Condition Keys