7.5 KiB
Object Lifecycle Management
minio
Table of Contents
Use MinIO Object Lifecycle Management to create rules for time or date based automatic transition or expiry of objects. For object transition, MinIO automatically moves the object to a configured remote storage tier. For object expiry, MinIO automatically deletes the object.
MinIO derives it's behavior and syntax from S3 lifecycle <object-lifecycle-mgmt.html>
for compatibility in migrating workloads and lifecycle rules from S3 to
MinIO. For example, you can export S3 lifecycle management rules and
import them into MinIO or vice-versa. MinIO uses JSON to describe
lifecycle management rules and may require conversion to or from XML as
part of importing S3 lifecycle rules.
Object Transition ("Tiering")
MinIO supports creating object transition lifecycle management rules, where MinIO can automatically move an object to a remote storage "tier". MinIO supports any of the following remote tier targets:
MinIO <minio-lifecycle-management-transition-to-minio>Amazon S3 <minio-lifecycle-management-transition-to-s3>Google Cloud Storage <minio-lifecycle-management-transition-to-gcs>Microsoft Azure Blob Storage <minio-lifecycle-management-transition-to-azure>
MinIO object transition supports use cases like moving aged data from MinIO clusters in private or public cloud infrastructure to low-cost private or public cloud storage solutions. MinIO manages retrieving tiered objects on-the-fly without any additional application-side logic.
Use the mc ilm tier add command to create a remote target for
tiering data to that target. You can then use the mc ilm rule add --transition-days command to
transition objects to that tier after a specified number of calendar
days.
RELEASE.2022-11-10T18-20-21Z
You can verify the tiering status of an object using mc ls against the bucket or
bucket prefix. The output includes the storage tier of each object:
$ mc ls play/mybucket
[2022-11-08 11:30:24 PST] 52MB STANDARD log-data.csv
[2022-11-09 12:20:18 PST] 120MB WARM event-2022-11-09.mp4STANDARDmarks objects stored on the MinIO deployment.WARMmarks objects stored on the remote tier with matching name.
Important
MinIO Object Transition supports cost-saving strategies around moving older or aged data to cost-optimized remote storage tiers, such as cloud storage or high-density HDD storage.
MinIO Object Transition does not provide backup and recovery functionality. You cannot use the remote tier as a recovery source in the event of data loss in MinIO.
Use either site replication <minio-site-replication-overview>
or bucket replication <minio-bucket-replication> to
support backup/recovery or BC/DR (Business Continuity / Disaster Recovery)
requirements.
Exclusive Access to Remote Data
Availability of Remote Data
Versioned Buckets
MinIO adopts S3 behavior <intro-lifecycle-rules.html#intro-lifecycle-rules-actions>
for transition rules on versioned buckets <minio-bucket-versioning>.
Specifically, MinIO by default applies the transition operation to the
current object version.
To transition noncurrent object versions, specify the ~mc ilm rule add --noncurrent-transition-days and
~mc ilm rule add --noncurrent-transition-tier
options when creating the transition rule.
Object Expiration
MinIO lifecycle management supports expiring objects on a bucket.
Object "expiration" involves performing a DELETE operation
on the object. For example, you can create a lifecycle management rule
to expire any object older than 365 days.
Use mc ilm rule add --expire-days to expire objects
after a specified number of calendar days.
For buckets with replication <minio-bucket-replication>
configured, MinIO does not replicate objects deleted by a lifecycle
management expiration rule. See minio-replication-behavior-delete for more
information.
Versioned Buckets
MinIO adopts S3 behavior <intro-lifecycle-rules.html#intro-lifecycle-rules-actions>
for expiration rules on versioned buckets <minio-bucket-versioning>.
MinIO has two specific default behaviors for versioned buckets:
MinIO applies the expiration option to only the current object version by creating a
DeleteMarkeras is normal with versioned delete.To expire noncurrent object versions, specify the
~mc ilm rule add --noncurrent-expire-daysoption when creating the expiration rule.MinIO does not expire
DeleteMarkerseven if no other versions of that object exist.To expire delete markers when there are no remaining versions for that object, specify the
~mc ilm rule add --expire-delete-markeroption when creating the expiration rule.
Lifecycle Management Object Scanner
MinIO uses a built-in scanner to actively check objects against all
configured lifecycle management rules. The scanner is a low-priority
process that yields to high I/O (Input / Output) workloads to prevent performance
spikes triggered by rule timing. The scanner may therefore not detect an
object as eligible for a configured transition or expiration lifecycle
rule until after the lifecycle rule period has passed.
Scanner performance typically depends on the available node resources, the size of the cluster, and the complexity of bucket hierarchy (objects and prefixes). For example, a cluster that starts with 100TB of data that then grows to 200TB of data may require more time to scan the entire namespace of buckets and objects given the same hardware and workload. As the cluster or workload increases, scanner performance decreases as it yields more frequently to ensure priority of normal S3 operations.
Consider regularly checking cluster metrics, capacity, and resource usage to ensure the cluster hardware is scaling alongside cluster and workload growth:
minio-metrics-and-alerts
/administration/object-management/transition-objects-to-minio.rst /administration/object-management/transition-objects-to-s3.rst /administration/object-management/transition-objects-to-gcs.rst /administration/object-management/transition-objects-to-azure.rst /administration/object-management/create-lifecycle-management-expiration-rule.rst