docs/source/administration/object-management/transition-objects-to-gcs.rst

Transition Objects from MinIO to GCS

minio

Table of Contents

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Google Cloud Storage backend. This procedure supports use cases like moving aged data to low-cost public cloud storage solutions after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart <mc-install> for instructions on downloading and installing mc.

Use the mc alias command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions <minio-lifecycle-management-transition-to-gcs-permissions> for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

• s3:PutLifecycleConfiguration
• s3:GetLifecycleConfiguration

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

• admin:SetTier
• admin:ListTier

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

/extra/examples/LifecycleManagementAdmin.json

Required GCS Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the GCS (Google Cloud Storage) credentials provide read, write, list, and delete permissions for the remote bucket.

Refer to the GCS IAM permissions documentation for more complete guidance on configuring the required permissions.

Considerations

Lifecycle Management Object Scanner

MinIO uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules. See minio-lifecycle-management-scanner for more information.

Exclusive Access to Remote Data

Availability of Remote Data

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

2) Configure the Remote Storage Tier

Use the mc admin tier add command to add a new Google Cloud Storage service as the remote storage tier:

mc admin tier add gcs TARGET TIER_NAME \
   --endpoint https://HOSTNAME \
   --bucket BUCKET \
   --prefix PREFIX \
   --credentials-file CREDENTIALS \
   --region REGION

The example above uses the following arguments:

Argument	Description
TARGET <mc admin tier add TARGET>	The alias <mc alias> of the MinIO deployment on which to configure the GCS (Google Cloud Storage) remote tier.
TIER_NAME <mc admin tier add TIER_NAME>	The name to associate with the new GCS (Google Cloud Storage) remote storage tier. Specify the name in all-caps, e.g. GCS_TIER. This value is required in the next step.
HOSTNAME <mc admin tier add --endpoint>	The URL endpoint for the GCS (Google Cloud Storage) storage backend.
BUCKET <mc admin tier add --bucket>	The name of the bucket on the GCS (Google Cloud Storage) storage backend to which MinIO transitions objects.
PREFIX <mc admin tier add --prefix>	The optional bucket prefix within which MinIO transitions objects. MinIO stores all transitioned objects in the specified BUCKET under a unique per-deployment prefix value. Omit this argument to use only that value for isolating and organizing data within the remote storage. MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This prefix should provide a clear reference back to the source MinIO deployment to faciliate ease of operations related to diagnostics, maintenance, or disaster recovery.
CREDENTIALS <mc admin tier add --credentials-file>	The credential file for a user on the remote GCS tier. The specified user credentials must correspond to a GCS user with the required permissions <minio-lifecycle-management-transition-to-gcs-permissions-remote>.
REGION <mc admin tier add --region>	The GCS (Google Cloud Storage) region of the specified BUCKET. You can safely omit this option if the HOSTNAME includes the region.

3) Create and Apply the Transition Rule

4) Verify the Transition Rule

Use the mc ilm ls command to review the configured transition rules:

mc ilm ls ALIAS/PATH --transition

• Replace ALIAS <mc ilm ls ALIAS> with the alias <mc alias> of the MinIO deployment.
• Replace PATH <mc ilm ls ALIAS> with the name of the bucket for which to retrieve the configured lifecycle management rules.