minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-28 19:42:10 +03:00
Code Activity

DOCS-699: improve documentation on storage classes for remote targets (#700)

Browse Source 
Closes #699
This commit is contained in:
Ravind Kumar
2023-01-12 15:14:13 -05:00
committed by GitHub
parent ea489b4a30
commit 3559267dea
8 changed files with 376 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
source/administration/object-management/object-lifecycle-management.rst
View File

@ -26,7 +26,7 @@ Object Transition ("Tiering")
MinIO supports creating object transition lifecycle management rules, where MinIO can automatically move an object to a remote storage "tier".
MinIO supports any of the following remote tier targets:
- :ref:`MinIO or other S3-compatible storage <minio-lifecycle-management-transition-to-s3>`
- :ref:`MinIO <minio-lifecycle-management-transition-to-minio>`
- :ref:`Amazon S3 <minio-lifecycle-management-transition-to-s3>`
- :ref:`Google Cloud Storage <minio-lifecycle-management-transition-to-gcs>`
- :ref:`Microsoft Azure Blob Storage
@ -126,6 +126,7 @@ Consider regularly checking cluster metrics, capacity, and resource usage to ens
.. toctree::
:hidden:
/administration/object-management/transition-objects-to-minio.rst
/administration/object-management/transition-objects-to-s3.rst
/administration/object-management/transition-objects-to-gcs.rst
/administration/object-management/transition-objects-to-azure.rst

57
source/administration/object-management/transition-objects-to-azure.rst
View File

@ -68,13 +68,25 @@ Required Azure Permissions
Object transition lifecycle management rules require additional permissions on
the remote storage tier. Specifically, MinIO requires the
:abbr:`Azure (Microsoft Azure)` credentials provide read, write, list, and
delete permissions for the remote bucket.
delete permissions for the remote storage account and container.
Refer to the `Azure RBAC
<https://docs.microsoft.com/en-us/azure/role-based-access-control/>`__
documentation for more complete guidance on configuring the required
permissions.
Remote Storage Account and Container Must Exist
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create the remote :azure-docs:`Azure storage account <storage/common/storage-account-overview>` and container *prior* to configuring lifecycle management tiers or rules using that resource as the target.
When :azure-docs:`creating the Azure storage account <storage/common/storage-account-create>`, ensure the storage account corresponds to either Standard or Premium blob storage with the locally redundant storage (LRS) redundancy option.
The Azure Go SDK API used by MinIO does not support any other redundancy options.
If you set a Storage Account :azure-docs:`default access tier <storage/blobs/access-tiers-online-manage>`, MinIO uses that default *if* you do not specify a :mc-cmd:`storage class <mc ilm tier add --storage-class>` when defining the remote tier.
Ensure you document the settings of both your Azure storage account and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes.
For more information on Azure storage accounts, see :azure-docs:`Storage accounts <storage/common/storage-account-overview#types-of-storage-accounts>`.
Considerations
--------------
@ -125,8 +137,9 @@ Use the :mc:`mc ilm tier add` command to add a new remote storage tier:
mc ilm tier add azure TARGET TIER_NAME \
--account-name ACCOUNT \
--account-key KEY \
--bucket BUCKET \
--prefix PREFIX
--bucket CONTAINER \
--prefix PREFIX \
--storage-class STORAGE_CLASS
The example above uses the following arguments:
@ -148,12 +161,25 @@ The example above uses the following arguments:
remote storage tier. Specify the name in all-caps, e.g. ``AZURE_TIER``.
This value is required in the next step.
* - :mc-cmd:`BUCKET <mc ilm tier add --bucket>`
- The name of the bucket on the :abbr:`Azure (Microsoft Azure)` storage
* - :mc-cmd:`ACCOUNT <mc ilm tier add --account-name>`
- The :azure-docs:`Storage Account <storage/common/storage-account-overview>` to use as the remote storage resource.
You cannot change this account name after creating the tier.
* - :mc-cmd:`KEY <mc ilm tier add --account-key>`
- The corresponding shared account key for the specified ``ACCOUNT``.
The account key must have an assigned Azure policy with the required :ref:`permissions
<minio-lifecycle-management-transition-to-azure-permissions-remote>`.
See :azure-docs:`Managing storage account access keys <storage/common/storage-account-keys-manage>` for more information.
* - :mc-cmd:`CONTAINER <mc ilm tier add --bucket>`
- The name of the container on the :abbr:`Azure (Microsoft Azure)` storage
backend to which MinIO transitions objects.
* - :mc-cmd:`PREFIX <mc ilm tier add --prefix>`
- The optional bucket prefix within which MinIO transitions objects.
- The optional container prefix within which MinIO transitions objects.
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
unique per-deployment prefix value. Omit this argument to use only that
@ -162,20 +188,21 @@ The example above uses the following arguments:
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to faciliate ease of operations related to
source MinIO deployment to facilitate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`ACCOUNT <mc ilm tier add --account-name>`
- The account name MinIO uses to access the bucket. The account name
*must* correspond to an :abbr:`Azure (Microsoft Azure)` user with the
required :ref:`permissions
<minio-lifecycle-management-transition-to-azure-permissions-remote>`.
* - :mc-cmd:`STORAGE_CLASS <mc ilm tier add --storage-class>`
- The Azure access tier MinIO applies to objects transitioned to the Azure container.
You cannot change this account name after creating the tier.
MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request.
MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention.
* - :mc-cmd:`KEY <mc ilm tier add --account-key>`
- The corresponding key for the specified ``ACCOUNT``.
The following Azure access tiers meet MinIO's requirements as a remote tier:
- ``Hot``
- ``Cool``
For more information, see :azure-docs:`Hot, cool, and archive access tiers for blob data <storage/blobs/access-tiers-overview.html>`.
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

43
source/administration/object-management/transition-objects-to-gcs.rst
View File

@ -75,6 +75,14 @@ Refer to the `GCS IAM permissions
documentation for more complete guidance on configuring the required
permissions.
Remote Bucket Must Exist
~~~~~~~~~~~~~~~~~~~~~~~~
Create the remote GCS bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target.
If you set a default GCS :gcs-docs:`storage class <storage-classes>`, MinIO uses that default *if* you do not specify a :mc-cmd:`storage class <mc ilm tier add --storage-class>` when defining the remote tier.
Ensure you document the settings of both your GCS bucket and MinIO tiering configuration to avoid any potential confusion, misconfiguration, or other unexpected outcomes.
Considerations
--------------
@ -124,7 +132,8 @@ service as the remote storage tier:
mc ilm tier add gcs TARGET TIER_NAME \
--bucket BUCKET \
--prefix PREFIX \
--credentials-file CREDENTIALS
--credentials-file CREDENTIALS \
--storage-class STORAGE_CLASS
The example above uses the following arguments:
@ -145,14 +154,6 @@ The example above uses the following arguments:
remote storage tier. Specify the name in all-caps, e.g. ``GCS_TIER``.
This value is required in the next step.
* - :mc-cmd:`CREDENTIALS <mc ilm tier add --credentials-file>`
- The `credential file
<https://cloud.google.com/docs/authentication/getting-started>`__ for a
user on the remote GCS tier. The specified user credentials *must*
correspond to a GCS user with the required
:ref:`permissions
<minio-lifecycle-management-transition-to-gcs-permissions-remote>`.
* - :mc-cmd:`BUCKET <mc ilm tier add --bucket>`
- The name of the bucket on the :abbr:`GCS (Google Cloud Storage)` storage
backend to which MinIO transitions objects.
@ -167,9 +168,31 @@ The example above uses the following arguments:
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to faciliate ease of operations related to
source MinIO deployment to facilitate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`CREDENTIALS <mc ilm tier add --credentials-file>`
- The `credential file
<https://cloud.google.com/docs/authentication/getting-started>`__ for a
user on the remote GCS tier. The specified user credentials *must*
correspond to a GCS user with the required
:ref:`permissions
<minio-lifecycle-management-transition-to-gcs-permissions-remote>`.
* - :mc-cmd:`STORAGE_CLASS <mc ilm tier add --storage-class>`
- The :abbr:`GCS (Google Cloud Storage)` storage class MinIO applies to objects transitioned to the GCS bucket.
MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request.
MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention.
The following GCS storage classes meet MinIO's requirements as a remote tier:
- ``STANDARD``
- ``NEARLINE``
- ``COLDLINE``
For more information, see :gcs-docs:`GCS storage class <storage-classes>`.
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

228
source/administration/object-management/transition-objects-to-minio.rst Normal file
View File

@ -0,0 +1,228 @@
.. _minio-lifecycle-management-transition-to-minio:
=============================================
Transition Objects to Remote MinIO Deployment
=============================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 2
The procedure on this page creates a new object lifecycle management rule that transitions objects from a bucket on a primary MinIO deployment to a bucket on a remote MinIO deployment.
This procedure supports cost-management strategies such as tiering objects from a "hot" MinIO deployment using NVMe storage to a "warm" MinIO deployment using SSD.
.. todo: diagram
Requirements
------------
Install and Configure ``mc``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on the MinIO cluster.
Install :mc:`mc` on a machine with network access to both source and destination
clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for
instructions on downloading and installing ``mc``.
Use the :mc:`mc alias` command to create an alias for the source MinIO cluster.
Alias creation requires specifying an access key for a user on the source and
destination clusters. The specified users must have :ref:`permissions
<minio-lifecycle-management-transition-to-minio-permissions>` for configuring and
applying transition operations.
.. _minio-lifecycle-management-transition-to-minio-permissions:
Required Source MinIO Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO requires the following permissions scoped to the bucket or buckets
for which you are creating lifecycle management rules.
- :policy-action:`s3:PutLifecycleConfiguration`
- :policy-action:`s3:GetLifecycleConfiguration`
MinIO also requires the following administrative permissions on the cluster
in which you are creating remote tiers for object transition lifecycle
management rules:
- :policy-action:`admin:SetTier`
- :policy-action:`admin:ListTier`
For example, the following policy provides permission for configuring object
transition lifecycle management rules on any bucket in the cluster:
.. literalinclude:: /extra/examples/LifecycleManagementAdmin.json
:language: json
:class: copyable
.. _minio-lifecycle-management-transition-to-minio-permissions-remote:
Required Remote MinIO Permissions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Object transition lifecycle management rules require additional permissions
on the remote storage tier. Specifically, MinIO requires the remote
tier credentials provide read, write, list, and delete permissions for the
remote bucket.
For example, the following policy on the remote MinIO deployment provides the necessary permission
for transitioning objects into and out of the remote tier:
.. literalinclude:: /extra/examples/LifecycleManagementUser.json
:language: json
:class: copyable
Modify the ``Resource`` for the bucket into which MinIO tiers objects.
Refer to the :ref:`minio-policy` documentation for more complete guidance on configuring the required permissions.
Remote Bucket Must Exist
~~~~~~~~~~~~~~~~~~~~~~~~
Create the remote bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target.
If the remote bucket contains existing data, use the :mc-cmd:`prefix <mc ilm tier add --prefix>` feature to isolate transitioned objects from any other objects on that bucket.
Considerations
--------------
Lifecycle Management Object Scanner
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO uses a scanner process to check objects against all configured
lifecycle management rules. Slow scanning due to high IO workloads or
limited system resources may delay application of lifecycle management
rules. See :ref:`minio-lifecycle-management-scanner` for more information.
Exclusive Access to Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-bucket-access-desc
:end-before: end-transition-bucket-access-desc
Availability of Remote Data
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-transition-data-loss-desc
:end-before: end-transition-data-loss-desc
Procedure
---------
1) Configure User Accounts and Policies for Lifecycle Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. |permissions| replace:: :ref:`permissions <minio-lifecycle-management-transition-to-minio-permissions>`
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-user-desc
:end-before: end-create-transition-user-desc
2) Configure the Remote Storage Tier
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc:`mc ilm tier add` command to add the remote MinIO deployment as the
new remote storage tier:
.. code-block:: shell
:class: copyable
mc ilm tier add minio TARGET TIER_NAME \
--endpoint https://HOSTNAME \
--access-key ACCESS_KEY \
--secret-key SECRET_KEY \
--bucket BUCKET \
--prefix PREFIX \
--storage-class STORAGE_CLASS \
--region REGION
The example above uses the following arguments:
.. list-table::
:header-rows: 1
:widths: 30 70
:width: 100%
* - Argument
- Description
* - :mc-cmd:`ALIAS <mc ilm tier add TARGET>`
- The :mc:`alias <mc alias>` of the MinIO deployment on which to configure
the MinIO remote tier.
* - :mc-cmd:`TIER_NAME <mc ilm tier add TIER_NAME>`
- The name to associate with the new MinIO remote storage tier. Specify the
name in all-caps, e.g. ``MINIO_WARM_TIER``. This value is required in the next
step.
* - :mc-cmd:`HOSTNAME <mc ilm tier add --endpoint>`
- The URL endpoint for the MinIO storage backend.
* - :mc-cmd:`ACCESS_KEY <mc ilm tier add --access-key>`
- The access key MinIO uses to access the bucket. The
access key *must* correspond to an IAM user with the
required
:ref:`permissions
<minio-lifecycle-management-transition-to-minio-permissions-remote>`.
* - :mc-cmd:`SECRET_KEY <mc ilm tier add --secret-key>`
- The corresponding secret key for the specified ``ACCESS_KEY``.
* - :mc-cmd:`BUCKET <mc ilm tier add --bucket>`
- The name of the bucket on the remote MinIO deployment to which the ``SOURCE``
transitions objects.
* - :mc-cmd:`PREFIX <mc ilm tier add --prefix>`
- The optional bucket prefix within which MinIO transitions objects.
MinIO stores all transitioned objects in the specified ``BUCKET`` under a
unique per-deployment prefix value. Omit this argument to use only that
value for isolating and organizing data within the remote storage.
MinIO recommends specifying this optional prefix for remote storage tiers
which contain other data, including transitioned objects from other MinIO
deployments. This prefix should provide a clear reference back to the
source MinIO deployment to facilitate ease of operations related to
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`STORAGE_CLASS <mc ilm tier add --storage-class>`
- The :ref:`Erasure Coding storage class <minio-ec-storage-class>` MinIO applies to objects transitions to the remote MinIO bucket.
Specify one of the following supported storage classes:
- ``STANDARD`` *Recommended*
- ``REDUCED``
* - :mc-cmd:`REGION <mc ilm tier add --region>`
- The MinIO region of the specified ``BUCKET``.
MinIO deployments typically do not require setting a region as part of setup.
Only include this option if you explicitly set the ``MINIO_SITE_REGION`` configuration setting for the deployment.
3) Create and Apply the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-minio-tiering.rst
:start-after: start-create-transition-rule-desc
:end-before: end-create-transition-rule-desc
4) Verify the Transition Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc:`mc ilm rule ls` command to review the configured transition
rules:
.. code-block:: shell
:class: copyable
mc ilm rule ls ALIAS/PATH --transition
- Replace :mc-cmd:`ALIAS <mc ilm rule ls ALIAS>` with the :mc:`alias <mc alias>`
of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc ilm rule ls ALIAS>` with the name of the bucket for
which to retrieve the configured lifecycle management rules.

21
source/administration/object-management/transition-objects-to-s3.rst
View File

@ -84,6 +84,11 @@ Refer to the :aws-docs:`Amazon S3 Permissions
documentation for more complete guidance on configuring the required
permissions.
Remote Bucket Must Exist
~~~~~~~~~~~~~~~~~~~~~~~~
Create the remote S3 bucket *prior* to configuring lifecycle management tiers or rules using that bucket as the target.
Considerations
--------------
@ -189,11 +194,21 @@ The example above uses the following arguments:
diagnostics, maintenance, or disaster recovery.
* - :mc-cmd:`STORAGE_CLASS <mc ilm tier add --storage-class>`
- The S3 storage class to which MinIO transitions objects. Specify
one of the following supported storage classes:
- The S3 storage class to which MinIO transitions objects.
MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request.
MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention.
The following S3 storage classes meet MinIO's requirements as a remote tier:
- ``STANDARD``
- ``REDUCED``
- ``STANDARD_IA``
- ``STANDARD_ONEZONE``
Omit this value to use the default storage class for the bucket.
Specifying this value overrides the bucket storage class.
For more information, see :s3-docs:`Using Amazon S3 storage classes <storage-class-intro.html>`.
* - :mc-cmd:`REGION <mc ilm tier add --region>`
- The AWS S3 region of the specified ``BUCKET``. You can safely omit this

1
source/default-conf.py
View File

@ -72,6 +72,7 @@ extlinks = {
'minio-docs' : ('https://min.io/docs/%s?ref=docs-internal',''),
'gke-docs' : ('https://cloud.google.com/kubernetes-engine/docs/%s',''),
'gcp-docs' : ('https://cloud.google.com/compute/docs/%s',''),
'gcs-docs' : ('https://cloud.google.com/storage/docs/%s',''),
'aks-docs' : ('https://learn.microsoft.com/en-us/azure/aks/%s',''),
'azure-docs' : ('https://learn.microsoft.com/en-us/azure/%s',''),

8
source/includes/common-minio-tiering.rst
View File

@ -111,9 +111,11 @@ This tutorial includes the necessary syntax for setting this prefix.
.. start-transition-data-loss-desc
MinIO creates metadata for each transitioned object that identifies its location
on the remote storage. This metadata is required for accessing the object, such
that applications cannot access a transition object independent of MinIO.
MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request.
MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention.
MinIO creates metadata for each transitioned object that identifies its location on the remote storage.
Applications cannot trivially identify and access a transitioned object independent of MinIO.
Availability of the transitioned data therefore depends on the same core
protections that :ref:`erasure coding <minio-erasure-coding>` and distributed
deployment topologies provide for all objects on the MinIO deployment. Using

60
source/reference/minio-mc/mc-ilm-tier-add.rst
View File

@ -194,19 +194,20 @@ The command accepts the following arguments:
.. mc-cmd:: --account-name
:optional:
The account name for a user on the remote Azure tier.
The user must have permission to perform read/write/list/delete operations on the remote bucket or bucket prefix.
The :azure-docs:`Storage Account <storage/common/storage-account-overview>` to use as the remote storage resource.
Required if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``azure``.
This option has no effect for any other value of ``TIER_TYPE``.
MinIO does *not* support changing the account name associated to an Azure remote tier.
Azure storage backends are tied to the account, such that changing the account would change the storage backend and prevent access to any objects transitioned to the original account/backend.
MinIO does *not* support changing the storage account name associated to an Azure remote tier.
Azure storage backends are tied to the storage account, such that changing this value would change the storage backend and prevent access to any objects transitioned to the original account/backend.
.. mc-cmd:: --account-key
:optional:
The account key for the :mc-cmd:`~mc ilm tier add --account-name` associated to the remote Azure tier.
The corresponding shared account key for the :mc-cmd:`~mc ilm tier add --account-name` associated to the remote Azure tier.
The account key must have an assigned Azure policy with the required :ref:`permissions <minio-lifecycle-management-transition-to-azure-permissions-remote>`.
Required if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``azure``.
This option has no effect for any other value of ``TIER_TYPE``.
@ -225,6 +226,8 @@ The command accepts the following arguments:
The bucket on the remote tier to which MinIO transitions objects.
For ``azure`` remote tiers, this value corresponds to the :azure-docs:`Container name <storage/blobs/storage-blobs-introduction#containers>`
.. mc-cmd:: --prefix
:optional:
@ -235,16 +238,47 @@ The command accepts the following arguments:
.. mc-cmd:: --storage-class
:optional:
The AWS storage class to use for objects transitioned by MinIO.
MinIO supports only the following storage classes:
The storage class ("access tier" for Microsoft Azure) MinIO applies to objects transitioned to the remote bucket.
- ``STANDARD``
- ``REDUCED_REDUNDANCY``
The storage class to apply to objects transitioned by MinIO to the remote bucket.
MinIO tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request.
MinIO therefore *cannot* support remote storage which requires rehydration, wait periods, or manual intervention.
Select the tab corresponding to the ``TIER_TYPE`` for a list of supported values for each tier:
Defaults to ``STANDARD`` if omitted.
.. tab-set::
This option only applies if :mc-cmd:`~mc ilm tier add TIER_TYPE` is ``s3`` or ``minio``.
This option has no effect for any other value of ``TIER_TYPE``.
.. tab-item:: minio
- ``STANDARD`` *Recommended*
- ``REDUCED``
For more information, see :ref:`Erasure Coding storage class <minio-ec-storage-class>`.
.. tab-item:: s3
- ``STANDARD``
- ``STANDARD_IA``
- ``ONEZONE_IA``
For more information, see :s3-docs:`Using Amazon S3 storage classes <storage-class-intro.html>`.
.. tab-item:: gcs
- ``STANDARD``
- ``NEARLINE``
- ``COLDLINE``
For more information, see :gcs-docs:`GCS storage class <storage-classes>`.
.. tab-item:: azure
- ``Hot``
- ``Cool``
For more information, see :azure-docs:`Hot, cool, and archive access tiers for blob data <storage/blobs/access-tiers-overview.html>`.
If omitted, objects use the default storage class defined for the remote bucket.
.. mc-cmd:: --region
:optional: