From bce5f8267fb547a8e5eed9b31fbd664a5ac90cef Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 22 Nov 2022 15:08:35 -0600 Subject: [PATCH] Updates for minio RELEASE.2022-08-22T23-53-06Z (#648) Doc updates related to for minio RELEASE.2022-08-22T23-53-06Z - Adds note about idempotent delete markers, which diverges from standard S3 - Adds info about queueing and re-attempting failed bucket replication tasks - Cleans up `mc ilm add` examples throughout the docs that did not reflect recent flag name changes - Adds a missing metric to Metrics and Alerts Closes #537 --- source/administration/bucket-replication.rst | 3 +++ ...eate-lifecycle-management-expiration-rule.rst | 16 ++++++++-------- .../object-lifecycle-management.rst | 10 +++++----- .../object-management/object-versioning.rst | 12 ++++++++++++ source/includes/common-minio-tiering.rst | 12 ++++++------ .../operations/monitoring/metrics-and-alerts.rst | 5 +++++ source/reference/minio-mc/mc-ilm-add.rst | 16 ++++++++-------- 7 files changed, 47 insertions(+), 27 deletions(-) diff --git a/source/administration/bucket-replication.rst b/source/administration/bucket-replication.rst index ef015888..b514bcf3 100644 --- a/source/administration/bucket-replication.rst +++ b/source/administration/bucket-replication.rst @@ -256,6 +256,9 @@ workers operating on that queue. MinIO continuously works to replicate and remove objects from the queue while scanning for new unreplicated objects to add to the queue. +MinIO queues failed replication operations and retries those operations until replication succeeds. +This helps keep replication up-to-date without relying on the MinIO scanner to notice unreplicated object versions. + .. versionchanged:: RELEASE.2022-08-11T04-37-28Z Failed or pending replications requeue automatically when performing a list or any ``GET`` or ``HEAD`` API method. diff --git a/source/administration/object-management/create-lifecycle-management-expiration-rule.rst b/source/administration/object-management/create-lifecycle-management-expiration-rule.rst index a3bdb738..1b849ec5 100644 --- a/source/administration/object-management/create-lifecycle-management-expiration-rule.rst +++ b/source/administration/object-management/create-lifecycle-management-expiration-rule.rst @@ -62,13 +62,13 @@ transition lifecycle management rules on any bucket in the cluster:. Expire Objects after Number of Days ----------------------------------- -Use :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --expiry-days` to +Use :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --expire-days` to expire bucket contents a number of days after object creation: .. code-block:: shell :class: copyable - mc ilm add ALIAS/PATH --expiry-days "DAYS" + mc ilm add ALIAS/PATH --expire-days "DAYS" - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. @@ -76,7 +76,7 @@ expire bucket contents a number of days after object creation: - Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. -- Replace :mc-cmd:`DAYS ` with the number of days after +- Replace :mc-cmd:`DAYS ` with the number of days after which to expire the object. For example, specify ``30`` to expire the object 30 days after creation. @@ -87,17 +87,17 @@ Use :mc:`mc ilm add` to expiring noncurrent object versions and object delete markers: - To expire noncurrent object versions after a specific duration in days, - include :mc-cmd:`~mc ilm add --noncurrentversion-expiration-days`. + include :mc-cmd:`~mc ilm add --noncurrent-expire-days`. - To expire delete markers for objects with no remaining versions, - include :mc-cmd:`~mc ilm add --expired-object-delete-marker`. + include :mc-cmd:`~mc ilm add --expire-delete-marker`. .. code-block:: shell :class: copyable mc ilm add ALIAS/PATH \ - --noncurrentversion-expiration-days NONCURRENT_DAYS \ - --expired-object-delete-marker + --noncurrent-expire-days NONCURRENT_DAYS \ + --expire-delete-marker - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. @@ -106,7 +106,7 @@ delete markers: S3-compatible host. - Replace :mc-cmd:`NONCURRENT_DAYS - ` with the number of days after + ` with the number of days after which to expire noncurrent object versions. For example, specify ``30d`` to expire a version after it has been noncurrent for at least 30 days. diff --git a/source/administration/object-management/object-lifecycle-management.rst b/source/administration/object-management/object-lifecycle-management.rst index 896a036b..e67ca399 100644 --- a/source/administration/object-management/object-lifecycle-management.rst +++ b/source/administration/object-management/object-lifecycle-management.rst @@ -69,8 +69,8 @@ on :ref:`versioned buckets `. Specifically, MinIO by default applies the transition operation to the *current* object version. To transition noncurrent object versions, specify the -:mc-cmd:`~mc ilm add --noncurrentversion-transition-days` and -:mc-cmd:`~mc ilm add --noncurrentversion-tier` options +:mc-cmd:`~mc ilm add --noncurrent-transition-days` and +:mc-cmd:`~mc ilm add --noncurrent-transition-tier` options when creating the transition rule. .. _minio-lifecycle-management-expiration: @@ -85,7 +85,7 @@ older than 365 days. .. todo: Diagram of MinIO Expiration -Use :mc-cmd:`mc ilm add --expiry-days` to expire objects after a specified +Use :mc-cmd:`mc ilm add --expire-days` to expire objects after a specified number of calendar days. For buckets with :ref:`replication ` configured, MinIO @@ -104,14 +104,14 @@ specific default behaviors for versioned buckets: creating a ``DeleteMarker`` as is normal with versioned delete. To expire noncurrent object versions, specify the - :mc-cmd:`~mc ilm add --noncurrentversion-expiration-days` option + :mc-cmd:`~mc ilm add --noncurrent-expire-days` option when creating the expiration rule. - MinIO does not expire ``DeleteMarkers`` *even if* no other versions of that object exist. To expire delete markers when there are no remaining versions for that - object, specify the :mc-cmd:`~mc ilm add --expired-object-delete-marker` + object, specify the :mc-cmd:`~mc ilm add --expire-delete-marker` option when creating the expiration rule. .. _minio-lifecycle-management-scanner: diff --git a/source/administration/object-management/object-versioning.rst b/source/administration/object-management/object-versioning.rst index cbd9fdf2..6f727a57 100644 --- a/source/administration/object-management/object-versioning.rst +++ b/source/administration/object-management/object-versioning.rst @@ -143,6 +143,18 @@ return the ``DeleteMarker`` back as part of the response. Similarly, performing a ``LIST`` operation by default returns only objects which are *not* a ``DeleteMarker``. +.. admonition:: MinIO Implements Idempotent Delete Markers + :class: note + + .. versionchanged:: RELEASE.2022-08-22T23-53-06Z + + Standard S3 implementations can create multiple sequential delete markers for the same object when processing simple ``DeleteObject`` requests with no version identifier. + See the S3 docs for details on :s3-docs:`managing delete markers `` + + MinIO diverges from standard S3 implementation by avoiding this potential duplication of delete markers. + When processing a ``Delete`` request with no version identifier, MinIO creates at most one Delete Marker for the specified object. + MinIO **does not** share S3's behavior in creating multiple sequential delete markers. + To permanently delete an object version, perform the ``DELETE`` operation and specify the version ID of the object to delete. Versioned delete operations are **irreversible**. diff --git a/source/includes/common-minio-tiering.rst b/source/includes/common-minio-tiering.rst index 835c85c7..1c47db93 100644 --- a/source/includes/common-minio-tiering.rst +++ b/source/includes/common-minio-tiering.rst @@ -8,10 +8,10 @@ specified number of calendar days: :class: copyable mc ilm add ALIAS/BUCKET \ - --tier TIERNAME \ + --transition-tier TIERNAME \ --transition-days DAYS \ - --noncurrentversion-transition-days NONCURRENT_DAYS - --noncurrentversion-tier TIERNAME + --noncurrent-transition-days NONCURRENT_DAYS + --noncurrent-transition-tier TIERNAME The example above specifies the following arguments: @@ -31,20 +31,20 @@ The example above specifies the following arguments: - Specify the full path to the bucket for which you are creating the lifecycle management rule. - * - :mc-cmd:`TIERNAME ` + * - :mc-cmd:`TIERNAME ` - The remote storage tier to which MinIO transitions objects. Specify the remote storage tier name created in the previous step. If you want to transition noncurrent object versions to a distinct remote tier, specify a different tier name for - :mc-cmd:`~mc ilm add --noncurrentversion-tier`. + :mc-cmd:`~mc ilm add --noncurrent-transition-tier`. * - :mc-cmd:`DAYS ` - The number of calendar days after which MinIO marks an object as eligible for transition. Specify the number of days as an integer, e.g. ``30`` for 30 days. - * - :mc-cmd:`NONCURRENT_DAYS ` + * - :mc-cmd:`NONCURRENT_DAYS ` - The number of calendar days after which MinIO marks a noncurrent object version as eligible for transition. MinIO specifically measures the time since an object *became* non-current instead of the object diff --git a/source/operations/monitoring/metrics-and-alerts.rst b/source/operations/monitoring/metrics-and-alerts.rst index cde405d5..4d1357ec 100644 --- a/source/operations/monitoring/metrics-and-alerts.rst +++ b/source/operations/monitoring/metrics-and-alerts.rst @@ -97,6 +97,11 @@ Object and Bucket Metrics Total bucket size in bytes in a given bucket. You can identify the bucket using the ``{ bucket="STRING" }`` label. +.. metric:: minio_bucket_quota_total_bytes + + Total bucket quota size in bytes. + You can identify the bucket using the ``{ bucket="STRING" }`` label. + Replication Metrics ~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/minio-mc/mc-ilm-add.rst b/source/reference/minio-mc/mc-ilm-add.rst index 1e2556b3..faa2f956 100644 --- a/source/reference/minio-mc/mc-ilm-add.rst +++ b/source/reference/minio-mc/mc-ilm-add.rst @@ -96,7 +96,7 @@ Parameters .. code-block:: none - mc ilm add --prefix "meetingnotes/" myminio/mydata/ --expiry-days "90" + mc ilm add --prefix "meetingnotes/" myminio/mydata/ --expire-days "90" The command creates a rule that expires objects in the ``mydata`` bucket of the ``myminio`` ALIAS after 90 days for any object with the ``meetingnotes/`` prefix. @@ -298,30 +298,30 @@ Examples Expire All Bucket Contents After Number of Days ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --expiry-days` to mark bucket contents for expiration after a number of days pass from the object's creation: +Use :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --expire-days` to mark bucket contents for expiration after a number of days pass from the object's creation: .. code-block:: shell :class: copyable - mc ilm add ALIAS/PATH --expiry-days "DAYS" + mc ilm add ALIAS/PATH --expire-days "DAYS" - Replace :mc-cmd:`ALIAS ` with the :mc:`alias ` of the S3-compatible host. - Replace :mc-cmd:`PATH ` with the path to the bucket on the S3-compatible host. -- Replace :mc-cmd:`DATE ` with the number of days after which to expire the object. +- Replace :mc-cmd:`DATE ` with the number of days after which to expire the object. For example, specify ``30`` to expire the object 30 days after creation. Transition Non-Current Object Versions at a Prefix to a Different Tier ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use the :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --prefix` and :mc-cmd:`~mc ilm add --tier` to transition older non-current versions of an object to a different storage tier. +Use the :mc:`mc ilm add` with :mc-cmd:`~mc ilm add --prefix` and :mc-cmd:`~mc ilm add --transition-tier` to transition older non-current versions of an object to a different storage tier. .. code-block:: shell :class: copyable mc ilm add --prefix "doc/" --transition-days "90" --tier "MINIOTIER-1" \ - --noncurrentversion-transition-days "45" --noncurrentversion-tier "MINIOTIER-2" \ + --noncurrent-transition-days "45" --noncurrent-transition-tier "MINIOTIER-2" \ myminio/mybucket/ This command looks at the contents with the ``doc/`` prefix in the ``mybucket`` bucket on the ``myminio`` deployment. @@ -333,12 +333,12 @@ This command looks at the contents with the ``doc/`` prefix in the ``mybucket`` Expire All Objects at a Prefix, Retain Current Object Versions Longer Than Non-Current Object Versions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Use the :mc:`mc ilm add` command with :mc-cmd:`~mc ilm add --prefix`, :mc-cmd:`~mc ilm add --expiry-days`, and :mc-cmd:`~mc ilm add --noncurrentversion-expiration-days` to expire current and non-current versions of an object at different times. +Use the :mc:`mc ilm add` command with :mc-cmd:`~mc ilm add --prefix`, :mc-cmd:`~mc ilm add --expire-days`, and :mc-cmd:`~mc ilm add --noncurrent-expire-days` to expire current and non-current versions of an object at different times. .. code-block:: shell :class: copyable - mc ilm add --prefix "doc/" --expiry-days "300" --noncurrentversion-expiration-days "100" myminio/mybucket/ + mc ilm add --prefix "doc/" --expire-days "300" --noncurrent-expire-days "100" myminio/mybucket/ This command looks at the contents with the ``doc/`` prefix in the ``mybucket`` bucket on the ``myminio`` deployment.