diff --git a/source/administration/object-management/object-lifecycle-management.rst b/source/administration/object-management/object-lifecycle-management.rst index ac292b07..9c651048 100644 --- a/source/administration/object-management/object-lifecycle-management.rst +++ b/source/administration/object-management/object-lifecycle-management.rst @@ -26,7 +26,7 @@ Object Transition ("Tiering") MinIO supports creating object transition lifecycle management rules, where MinIO can automatically move an object to a remote storage "tier". MinIO supports any of the following remote tier targets: -- :ref:`MinIO or other S3-compatible storage ` +- :ref:`MinIO ` - :ref:`Amazon S3 ` - :ref:`Google Cloud Storage ` - :ref:`Microsoft Azure Blob Storage @@ -126,6 +126,7 @@ Consider regularly checking cluster metrics, capacity, and resource usage to ens .. toctree:: :hidden: + /administration/object-management/transition-objects-to-minio.rst /administration/object-management/transition-objects-to-s3.rst /administration/object-management/transition-objects-to-gcs.rst /administration/object-management/transition-objects-to-azure.rst diff --git a/source/administration/object-management/transition-objects-to-azure.rst b/source/administration/object-management/transition-objects-to-azure.rst index 296d75f2..2f7ec7d9 100644 --- a/source/administration/object-management/transition-objects-to-azure.rst +++ b/source/administration/object-management/transition-objects-to-azure.rst @@ -68,13 +68,25 @@ Required Azure Permissions Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the :abbr:`Azure (Microsoft Azure)` credentials provide read, write, list, and -delete permissions for the remote bucket. +delete permissions for the remote storage account and container. Refer to the `Azure RBAC `__ documentation for more complete guidance on configuring the required permissions. +Remote Storage Account and Container Must Exist +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Create the remote :azure-docs:`Azure storage account ` and container *prior* to configuring lifecycle management tiers or rules using that resource as the target. +When :azure-docs:`creating the Azure storage account `, ensure the storage account corresponds to either Standard or Premium blob storage with the locally redundant storage (LRS) redundancy option. +The Azure Go SDK API used by MinIO does not support any other redundancy options. + +If you set a Storage Account :azure-docs:`default access tier `, MinIO uses that default *if* you do not specify a :mc-cmd:`storage class ` when defining the remote tier. +Ensure you document the settings of both your Azure storage account and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes. + +For more information on Azure storage accounts, see :azure-docs:`Storage accounts `. + Considerations -------------- @@ -125,8 +137,9 @@ Use the :mc:`mc ilm tier add` command to add a new remote storage tier: mc ilm tier add azure TARGET TIER_NAME \ --account-name ACCOUNT \ --account-key KEY \ - --bucket BUCKET \ - --prefix PREFIX + --bucket CONTAINER \ + --prefix PREFIX \ + --storage-class STORAGE_CLASS The example above uses the following arguments: @@ -148,12 +161,25 @@ The example above uses the following arguments: remote storage tier. Specify the name in all-caps, e.g. ``AZURE_TIER``. This value is required in the next step. - * - :mc-cmd:`BUCKET ` - - The name of the bucket on the :abbr:`Azure (Microsoft Azure)` storage + * - :mc-cmd:`ACCOUNT ` + - The :azure-docs:`Storage Account ` to use as the remote storage resource. + + You cannot change this account name after creating the tier. + + * - :mc-cmd:`KEY ` + - The corresponding shared account key for the specified ``ACCOUNT``. + + The account key must have an assigned Azure policy with the required :ref:`permissions + `. + + See :azure-docs:`Managing storage account access keys ` for more information. + + * - :mc-cmd:`CONTAINER ` + - The name of the container on the :abbr:`Azure (Microsoft Azure)` storage backend to which MinIO transitions objects. * - :mc-cmd:`PREFIX ` - - The optional bucket prefix within which MinIO transitions objects. + - The optional container prefix within which MinIO transitions objects. MinIO stores all transitioned objects in the specified ``BUCKET`` under a unique per-deployment prefix value. Omit this argument to use only that @@ -162,20 +188,21 @@ The example above uses the following arguments: MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This prefix should provide a clear reference back to the - source MinIO deployment to faciliate ease of operations related to + source MinIO deployment to facilitate ease of operations related to diagnostics, maintenance, or disaster recovery. - * - :mc-cmd:`ACCOUNT ` - - The account name MinIO uses to access the bucket. The account name - *must* correspond to an :abbr:`Azure (Microsoft Azure)` user with the - required :ref:`permissions - `. + * - :mc-cmd:`STORAGE_CLASS ` + - The Azure access tier MinIO applies to objects transitioned to the Azure container. - You cannot change this account name after creating the tier. + MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. + MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention. - * - :mc-cmd:`KEY ` - - The corresponding key for the specified ``ACCOUNT``. + The following Azure access tiers meet MinIO's requirements as a remote tier: + - ``Hot`` + - ``Cool`` + + For more information, see :azure-docs:`Hot, cool, and archive access tiers for blob data `. 3) Create and Apply the Transition Rule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/administration/object-management/transition-objects-to-gcs.rst b/source/administration/object-management/transition-objects-to-gcs.rst index 7d3a3119..cb35dc84 100644 --- a/source/administration/object-management/transition-objects-to-gcs.rst +++ b/source/administration/object-management/transition-objects-to-gcs.rst @@ -75,6 +75,14 @@ Refer to the `GCS IAM permissions documentation for more complete guidance on configuring the required permissions. +Remote Bucket Must Exist +~~~~~~~~~~~~~~~~~~~~~~~~ + +Create the remote GCS bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target. + +If you set a default GCS :gcs-docs:`storage class `, MinIO uses that default *if* you do not specify a :mc-cmd:`storage class ` when defining the remote tier. +Ensure you document the settings of both your GCS bucket and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes. + Considerations -------------- @@ -124,7 +132,8 @@ service as the remote storage tier: mc ilm tier add gcs TARGET TIER_NAME \ --bucket BUCKET \ --prefix PREFIX \ - --credentials-file CREDENTIALS + --credentials-file CREDENTIALS \ + --storage-class STORAGE_CLASS The example above uses the following arguments: @@ -145,14 +154,6 @@ The example above uses the following arguments: remote storage tier. Specify the name in all-caps, e.g. ``GCS_TIER``. This value is required in the next step. - * - :mc-cmd:`CREDENTIALS ` - - The `credential file - `__ for a - user on the remote GCS tier. The specified user credentials *must* - correspond to a GCS user with the required - :ref:`permissions - `. - * - :mc-cmd:`BUCKET ` - The name of the bucket on the :abbr:`GCS (Google Cloud Storage)` storage backend to which MinIO transitions objects. @@ -167,9 +168,31 @@ The example above uses the following arguments: MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This prefix should provide a clear reference back to the - source MinIO deployment to faciliate ease of operations related to + source MinIO deployment to facilitate ease of operations related to diagnostics, maintenance, or disaster recovery. + * - :mc-cmd:`CREDENTIALS ` + - The `credential file + `__ for a + user on the remote GCS tier. The specified user credentials *must* + correspond to a GCS user with the required + :ref:`permissions + `. + + * - :mc-cmd:`STORAGE_CLASS ` + - The :abbr:`GCS (Google Cloud Storage)` storage class MinIO applies to objects transitioned to the GCS bucket. + + MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. + MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention. + + The following GCS storage classes meet MinIO's requirements as a remote tier: + + - ``STANDARD`` + - ``NEARLINE`` + - ``COLDLINE`` + + For more information, see :gcs-docs:`GCS storage class `. + 3) Create and Apply the Transition Rule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/administration/object-management/transition-objects-to-minio.rst b/source/administration/object-management/transition-objects-to-minio.rst new file mode 100644 index 00000000..1ea27a05 --- /dev/null +++ b/source/administration/object-management/transition-objects-to-minio.rst @@ -0,0 +1,228 @@ +.. _minio-lifecycle-management-transition-to-minio: + +============================================= +Transition Objects to Remote MinIO Deployment +============================================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +The procedure on this page creates a new object lifecycle management rule that transitions objects from a bucket on a primary MinIO deployment to a bucket on a remote MinIO deployment. +This procedure supports cost-management strategies such as tiering objects from a "hot" MinIO deployment using NVMe storage to a "warm" MinIO deployment using SSD. + +.. todo: diagram + +Requirements +------------ + +Install and Configure ``mc`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This procedure uses :mc:`mc` for performing operations on the MinIO cluster. +Install :mc:`mc` on a machine with network access to both source and destination +clusters. See the ``mc`` :ref:`Installation Quickstart ` for +instructions on downloading and installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for the source MinIO cluster. +Alias creation requires specifying an access key for a user on the source and +destination clusters. The specified users must have :ref:`permissions +` for configuring and +applying transition operations. + +.. _minio-lifecycle-management-transition-to-minio-permissions: + +Required Source MinIO Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO requires the following permissions scoped to the bucket or buckets +for which you are creating lifecycle management rules. + +- :policy-action:`s3:PutLifecycleConfiguration` +- :policy-action:`s3:GetLifecycleConfiguration` + +MinIO also requires the following administrative permissions on the cluster +in which you are creating remote tiers for object transition lifecycle +management rules: + +- :policy-action:`admin:SetTier` +- :policy-action:`admin:ListTier` + +For example, the following policy provides permission for configuring object +transition lifecycle management rules on any bucket in the cluster: + +.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json + :language: json + :class: copyable + +.. _minio-lifecycle-management-transition-to-minio-permissions-remote: + +Required Remote MinIO Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Object transition lifecycle management rules require additional permissions +on the remote storage tier. Specifically, MinIO requires the remote +tier credentials provide read, write, list, and delete permissions for the +remote bucket. + +For example, the following policy on the remote MinIO deployment provides the necessary permission +for transitioning objects into and out of the remote tier: + +.. literalinclude:: /extra/examples/LifecycleManagementUser.json + :language: json + :class: copyable + +Modify the ``Resource`` for the bucket into which MinIO tiers objects. + +Refer to the :ref:`minio-policy` documentation for more complete guidance on configuring the required permissions. + +Remote Bucket Must Exist +~~~~~~~~~~~~~~~~~~~~~~~~ + +Create the remote bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target. + +If the remote bucket contains existing data, use the :mc-cmd:`prefix ` feature to isolate transitioned objects from any other objects on that bucket. + +Considerations +-------------- + +Lifecycle Management Object Scanner +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO uses a scanner process to check objects against all configured +lifecycle management rules. Slow scanning due to high IO workloads or +limited system resources may delay application of lifecycle management +rules. See :ref:`minio-lifecycle-management-scanner` for more information. + +Exclusive Access to Remote Data +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-transition-bucket-access-desc + :end-before: end-transition-bucket-access-desc + +Availability of Remote Data +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-transition-data-loss-desc + :end-before: end-transition-data-loss-desc + +Procedure +--------- + +1) Configure User Accounts and Policies for Lifecycle Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. |permissions| replace:: :ref:`permissions ` + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-user-desc + :end-before: end-create-transition-user-desc + +2) Configure the Remote Storage Tier +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc:`mc ilm tier add` command to add the remote MinIO deployment as the +new remote storage tier: + +.. code-block:: shell + :class: copyable + + mc ilm tier add minio TARGET TIER_NAME \ + --endpoint https://HOSTNAME \ + --access-key ACCESS_KEY \ + --secret-key SECRET_KEY \ + --bucket BUCKET \ + --prefix PREFIX \ + --storage-class STORAGE_CLASS \ + --region REGION + +The example above uses the following arguments: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + :width: 100% + + * - Argument + - Description + + * - :mc-cmd:`ALIAS ` + - The :mc:`alias ` of the MinIO deployment on which to configure + the MinIO remote tier. + + * - :mc-cmd:`TIER_NAME ` + - The name to associate with the new MinIO remote storage tier. Specify the + name in all-caps, e.g. ``MINIO_WARM_TIER``. This value is required in the next + step. + + * - :mc-cmd:`HOSTNAME ` + - The URL endpoint for the MinIO storage backend. + + * - :mc-cmd:`ACCESS_KEY ` + - The access key MinIO uses to access the bucket. The + access key *must* correspond to an IAM user with the + required + :ref:`permissions + `. + + * - :mc-cmd:`SECRET_KEY ` + - The corresponding secret key for the specified ``ACCESS_KEY``. + + * - :mc-cmd:`BUCKET ` + - The name of the bucket on the remote MinIO deployment to which the ``SOURCE`` + transitions objects. + + * - :mc-cmd:`PREFIX ` + - The optional bucket prefix within which MinIO transitions objects. + + MinIO stores all transitioned objects in the specified ``BUCKET`` under a + unique per-deployment prefix value. Omit this argument to use only that + value for isolating and organizing data within the remote storage. + + MinIO recommends specifying this optional prefix for remote storage tiers + which contain other data, including transitioned objects from other MinIO + deployments. This prefix should provide a clear reference back to the + source MinIO deployment to facilitate ease of operations related to + diagnostics, maintenance, or disaster recovery. + + * - :mc-cmd:`STORAGE_CLASS ` + - The :ref:`Erasure Coding storage class ` MinIO applies to objects transitions to the remote MinIO bucket. + Specify one of the following supported storage classes: + + - ``STANDARD`` *Recommended* + - ``REDUCED`` + + * - :mc-cmd:`REGION ` + - The MinIO region of the specified ``BUCKET``. + + MinIO deployments typically do not require setting a region as part of setup. + Only include this option if you explicitly set the ``MINIO_SITE_REGION`` configuration setting for the deployment. + +3) Create and Apply the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. include:: /includes/common-minio-tiering.rst + :start-after: start-create-transition-rule-desc + :end-before: end-create-transition-rule-desc + + +4) Verify the Transition Rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :mc:`mc ilm rule ls` command to review the configured transition +rules: + +.. code-block:: shell + :class: copyable + + mc ilm rule ls ALIAS/PATH --transition + +- Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` + of the MinIO deployment. + +- Replace :mc-cmd:`PATH ` with the name of the bucket for + which to retrieve the configured lifecycle management rules. diff --git a/source/administration/object-management/transition-objects-to-s3.rst b/source/administration/object-management/transition-objects-to-s3.rst index 2b1d7db5..41ea7cbf 100644 --- a/source/administration/object-management/transition-objects-to-s3.rst +++ b/source/administration/object-management/transition-objects-to-s3.rst @@ -84,6 +84,11 @@ Refer to the :aws-docs:`Amazon S3 Permissions documentation for more complete guidance on configuring the required permissions. +Remote Bucket Must Exist +~~~~~~~~~~~~~~~~~~~~~~~~ + +Create the remote S3 bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target. + Considerations -------------- @@ -189,11 +194,21 @@ The example above uses the following arguments: diagnostics, maintenance, or disaster recovery. * - :mc-cmd:`STORAGE_CLASS ` - - The S3 storage class to which MinIO transitions objects. Specify - one of the following supported storage classes: + - The S3 storage class to which MinIO transitions objects. + + MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. + MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention. + + The following S3 storage classes meet MinIO's requirements as a remote tier: - ``STANDARD`` - - ``REDUCED`` + - ``STANDARD_IA`` + - ``STANDARD_ONEZONE`` + + Omit this value to use the default storage class for the bucket. + Specifying this value overrides the bucket storage class. + + For more information, see :s3-docs:`Using Amazon S3 storage classes `. * - :mc-cmd:`REGION ` - The AWS S3 region of the specified ``BUCKET``. You can safely omit this diff --git a/source/default-conf.py b/source/default-conf.py index b518674a..7c620960 100644 --- a/source/default-conf.py +++ b/source/default-conf.py @@ -72,6 +72,7 @@ extlinks = { 'minio-docs' : ('https://min.io/docs/%s?ref=docs-internal',''), 'gke-docs' : ('https://cloud.google.com/kubernetes-engine/docs/%s',''), 'gcp-docs' : ('https://cloud.google.com/compute/docs/%s',''), + 'gcs-docs' : ('https://cloud.google.com/storage/docs/%s',''), 'aks-docs' : ('https://learn.microsoft.com/en-us/azure/aks/%s',''), 'azure-docs' : ('https://learn.microsoft.com/en-us/azure/%s',''), diff --git a/source/includes/common-minio-tiering.rst b/source/includes/common-minio-tiering.rst index cc737dd7..95388780 100644 --- a/source/includes/common-minio-tiering.rst +++ b/source/includes/common-minio-tiering.rst @@ -111,9 +111,11 @@ This tutorial includes the necessary syntax for setting this prefix. .. start-transition-data-loss-desc -MinIO creates metadata for each transitioned object that identifies its location -on the remote storage. This metadata is required for accessing the object, such -that applications cannot access a transition object independent of MinIO. +MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. +MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention. + +MinIO creates metadata for each transitioned object that identifies its location on the remote storage. +Applications cannot trivially identify and access a transitioned object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that :ref:`erasure coding ` and distributed deployment topologies provide for all objects on the MinIO deployment. Using diff --git a/source/reference/minio-mc/mc-ilm-tier-add.rst b/source/reference/minio-mc/mc-ilm-tier-add.rst index be3ecaca..6c00bb19 100644 --- a/source/reference/minio-mc/mc-ilm-tier-add.rst +++ b/source/reference/minio-mc/mc-ilm-tier-add.rst @@ -194,19 +194,20 @@ The command accepts the following arguments: .. mc-cmd:: --account-name :optional: - The account name for a user on the remote Azure tier. - The user must have permission to perform read/write/list/delete operations on the remote bucket or bucket prefix. - + The :azure-docs:`Storage Account ` to use as the remote storage resource. + Required if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``azure``. This option has no effect for any other value of ``TIER_TYPE``. - MinIO does *not* support changing the account name associated to an Azure remote tier. - Azure storage backends are tied to the account, such that changing the account would change the storage backend and prevent access to any objects transitioned to the original account/backend. + MinIO does *not* support changing the storage account name associated to an Azure remote tier. + Azure storage backends are tied to the storage account, such that changing this value would change the storage backend and prevent access to any objects transitioned to the original account/backend. .. mc-cmd:: --account-key :optional: - The account key for the :mc-cmd:`~mc ilm tier add --account-name` associated to the remote Azure tier. + The corresponding shared account key for the :mc-cmd:`~mc ilm tier add --account-name` associated to the remote Azure tier. + + The account key must have an assigned Azure policy with the required :ref:`permissions `. Required if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``azure``. This option has no effect for any other value of ``TIER_TYPE``. @@ -225,6 +226,8 @@ The command accepts the following arguments: The bucket on the remote tier to which MinIO transitions objects. + For ``azure`` remote tiers, this value corresponds to the :azure-docs:`Container name ` + .. mc-cmd:: --prefix :optional: @@ -235,16 +238,47 @@ The command accepts the following arguments: .. mc-cmd:: --storage-class :optional: - The AWS storage class to use for objects transitioned by MinIO. - MinIO supports only the following storage classes: + The storage class ("access tier" for Microsoft Azure) MinIO applies to objects transitioned to the remote bucket. - - ``STANDARD`` - - ``REDUCED_REDUNDANCY`` + The storage class to apply to objects transitioned by MinIO to the remote bucket. + MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. + MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention. + + Select the tab corresponding to the ``TIER_TYPE`` for a list of supported values for each tier: - Defaults to ``STANDARD`` if omitted. + .. tab-set:: - This option only applies if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``s3`` or ``minio``. - This option has no effect for any other value of ``TIER_TYPE``. + .. tab-item:: minio + + - ``STANDARD`` *Recommended* + - ``REDUCED`` + + For more information, see :ref:`Erasure Coding storage class `. + + .. tab-item:: s3 + + - ``STANDARD`` + - ``STANDARD_IA`` + - ``ONEZONE_IA`` + + For more information, see :s3-docs:`Using Amazon S3 storage classes `. + + .. tab-item:: gcs + + - ``STANDARD`` + - ``NEARLINE`` + - ``COLDLINE`` + + For more information, see :gcs-docs:`GCS storage class `. + + .. tab-item:: azure + + - ``Hot`` + - ``Cool`` + + For more information, see :azure-docs:`Hot, cool, and archive access tiers for blob data `. + + If omitted, objects use the default storage class defined for the remote bucket. .. mc-cmd:: --region :optional: