From 4fc6df9c920a235642dcf7676583383dff05183e Mon Sep 17 00:00:00 2001 From: ravindk89 Date: Tue, 29 Jun 2021 18:22:35 -0400 Subject: [PATCH] DOCS-352: Add replica metadata sync, existing object replication Apply suggestions from code review Co-authored-by: Poorna Krishnamoorthy --- .../lifecycle-management-overview.rst | 4 + .../minio-cli/minio-mc/mc-replicate.rst | 188 ++++++++++++------ ...server-side-one-way-bucket-replication.rst | 38 ++-- ...server-side-two-way-bucket-replication.rst | 56 +++--- source/replication/replication-overview.rst | 116 ++++++++++- 5 files changed, 298 insertions(+), 104 deletions(-) diff --git a/source/lifecycle-management/lifecycle-management-overview.rst b/source/lifecycle-management/lifecycle-management-overview.rst index 748f17b4..1204a52c 100644 --- a/source/lifecycle-management/lifecycle-management-overview.rst +++ b/source/lifecycle-management/lifecycle-management-overview.rst @@ -92,6 +92,10 @@ options to create new expiration rules on a bucket: - :mc-cmd-option:`mc ilm add expiry-days` to expire objects after a specified number of calendar days. +For buckets with :ref:`replication ` configured, MinIO +does not replicate objects deleted by a lifecycle management expiration rule. +See :ref:`minio-replication-behavior-delete` for more information. + Versioned Buckets ~~~~~~~~~~~~~~~~~ diff --git a/source/reference/minio-cli/minio-mc/mc-replicate.rst b/source/reference/minio-cli/minio-mc/mc-replicate.rst index 4c68cf33..bee6ce9a 100644 --- a/source/reference/minio-cli/minio-mc/mc-replicate.rst +++ b/source/reference/minio-cli/minio-mc/mc-replicate.rst @@ -165,15 +165,49 @@ documentation on adding users and policies to a MinIO cluster. Replication of Existing Objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -MinIO performs replication as part of writing an object (PUT operations). -MinIO does *not* apply replication rules to existing objects written -*before* enabling replication. +Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z +` and :mc:`minio` +:minio-git:`RELEASE.2021-06-07T21-40-51Z +`, MinIO supports automatically +replicating existing objects in a bucket. MinIO existing object replication +implements functionality similar to +`AWS: Replicating existing objects between S3 buckets +`__ +without the overhead of contacting technical support. -For buckets with existing objects, consider using :mc:`mc mirror` or -:mc:`mc cp` to seed the destination bucket. Consider scheduling a maintenance -window during which applications stop writes to the bucket. Once -the :mc:`~mc mirror` or :mc:`~mc cp` fully sync the source and destination, -enable bucket replication and resume normal operations on the bucket. +- To enable replication of existing objects when creating a new replication + rule, include ``"existing-objects"`` to the list of replication features + specified to :mc-cmd-option:`mc replicate add replicate`. + +- To enable replication of existing objects for an existing replication rule, + add ``"existing-objects"`` to the list of existing replication features using + :mc-cmd-option:`mc replicate add replicate`. You must specify *all* + desired replication features when editing the replication rule. + +See :ref:`minio-replication-behavior-existing-objects` for more complete +documentation on this behavior. + +Synchronization of Metadata Changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO supports :ref:`two-way active-active +` replication configurations, where +MinIO synchronizes new and modified objects between a bucket on two MinIO +deployments. Starting with :mc:`mc` :minio-git:`RELEASE.2021-05-18T03-39-44Z +`, MinIO by default synchronizes +metadata-only changes to a replicated object back to the "source" deployment. +Prior to the this update, MinIO did not support synchronizing metadata-only +changes to a replicated object. + +With metadata synchronization enabled, MinIO resets the object +:ref:`replication status ` to indicate +replication eligibility. Specifically, when an application performs a +metadata-only update to an object with the ``REPLICA`` status, MinIO marks the +object as ``PENDING`` and eligible for replication. + +To disable metadata synchronization, use the +:mc-cmd-option:`mc replicate edit replicate` command and omit +``replica-metadata-sync`` from the replication feature list. Replication of Delete Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -184,52 +218,21 @@ Specifically, MinIO can replicate both of specific versioned objects: - For delete operations on an object, MinIO replication also creates the delete - marker on the target bucket. + marker on the target bucket. - For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket. +MinIO does *not* replicate objects deleted due to +:ref:`lifecycle management expiration rules +`. MinIO only replicates explicit +client-driven delete operations. + MinIO requires explicitly enabling replication of delete operations using the :mc-cmd-option:`mc replicate add replicate` flag. This procedure includes the required flags for enabling replication of delete operations and delete markers. - -.. note:: - - If a delete operation removes the last object in a bucket prefix, MinIO - recursively deletes any empty directories within that prefix. For example: - - ``play/mybucket/path/to/foo.txt`` - - The bucket prefix is ``/path/to/foo.txt``. If ``foo.txt`` is the last object - in prefix, MinIO deletes the entire prefix. If an object exists at ``/path``, - MinIO stops deleting the prefix at that point. - -.. admonition:: MinIO Trims Empty Object Prefixes - - If a delete operation removes the last object in a bucket prefix, MinIO - recursively removes each empty part of the prefix up to the bucket root. - MinIO only applies the recursive removal to prefixes created *implicitly* as - part of object write operations - that is, the prefix was not created using - an explicit directory creation command such as :mc:`mc mb`. - - If a replication rule enables replication delete operations, the replication - process *also* applies the implicit prefix trimming behavior on the - destination MinIO cluster. - - For example, consider a bucket ``photos`` with the following object prefixes: - - - ``photos/2021/january/myphoto.jpg`` - - ``photos/2021/february/myotherphoto.jpg`` - - ``photos/NYE21/NewYears.jpg`` - - ``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`. - All other prefixes were *implicitly* created as part of writing the object - located at that prefix. If a command removes ``myphoto.jpg``, it also - automatically trims the empty ``/janaury`` prefix. If then removes - the ``myotherphoto.jpg``, it also automatically trims both the ``/february`` - prefix *and* the now-empty ``/2021`` prefix. If removes the - ``NewYears.jpg`` object, the ``/NYE21`` prefix remains in place since it was - *explicitly* created. +See :ref:`minio-replication-behavior-delete` for more complete documentation +on this behavior. Replication of Encrypted Objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -448,7 +451,7 @@ Syntax *Optional* Specify a comma-separated list of the following values to enable extended - replication features: + replication features. - ``delete`` - Directs MinIO to replicate DELETE operations to the destination bucket. @@ -456,6 +459,8 @@ Syntax - ``delete-marker`` - Directs MinIO to replicate delete markers to the destination bucket. + - ``existing-objects`` - Directs MinIO to replicate objects created + before replication was enabled *or* while replication was suspended. .. mc-cmd:: tags :option: @@ -522,6 +527,15 @@ Syntax does not begin replicating objects using the rule until it is enabled using :mc-cmd:`mc replicate edit`. + Objects created while replication is disabled are not + immediately eligible for replication after enabling the rule. + You must explicitly enable replication of existing + objects by including ``"existing-objects"`` to the list of + replication features specified to + :mc-cmd-option:`mc replicate edit replicate`. See + :ref:`minio-replication-behavior-existing-objects` for more + information. + .. mc-cmd:: edit :fullpath: @@ -581,10 +595,21 @@ Syntax - ``delete-marker`` - Directs MinIO to replicate delete markers to the destination bucket. - MinIO does *not* apply the updated replication rules to existing - objects in the source bucket. For example, enabling delete marker - replication does not automatically replicate existing objects with - delete markers. + - ``replica-metadata-sync`` - Directs MinIO to synchronize metadata-only + changes on a replicated object back to the source. This feature only + effects :ref:`two-way active-active + ` replication + configurations. + + Omitting this value directs MinIO to stop replicating metadata-only + changes back to the source. + + - ``existing-objects`` - Directs MinIO to replicate objects created + prior to configuring or enabling replication. MinIO by default does + *not* synchronize existing objects to the remote target. + + See :ref:`minio-replication-behavior-existing-objects` for more + information. .. mc-cmd:: tags @@ -639,12 +664,17 @@ Syntax Enables or disables the replication rule. Specify one of the following values: - - ``"enable"`` - Enables the replication rule. MinIO begins replicating - objects written *after* enabling the rule. Existing objects require - manual migration to the destination bucket. + - ``"enable"`` - Enables the replication rule. - ``"disable"`` - Disables the replication rule. + Objects created while replication is disabled are not immediately eligible + for replication after enabling the rule. You must explicitly enable + replication of existing objects by including ``"existing-objects"`` to the + list of replication features specified to + :mc-cmd-option:`mc replicate edit replicate`. See + :ref:`minio-replication-behavior-existing-objects` for more information. + .. mc-cmd:: ls :fullpath: @@ -666,11 +696,11 @@ Syntax The full path to the bucket on which to list the replication configurations. Specify the :mc:`alias ` of a configured MinIO service as the prefix to the - :mc-cmd:`~mc replicate add SOURCE` path. For example: + :mc-cmd:`~mc replicate ls SOURCE` path. For example: .. code-block:: shell - mc replicate add play/mybucket + mc replicate ls play/mybucket .. mc-cmd:: insecure :option: @@ -765,11 +795,11 @@ Syntax The full path to the bucket to which to import the replication configurations. Specify the :mc:`alias ` of a configured MinIO service as the prefix to the - :mc-cmd:`~mc replicate add SOURCE` path. For example: + :mc-cmd:`~mc replicate import SOURCE` path. For example: .. code-block:: shell - mc replicate export play/mybucket + mc replicate import play/mybucket .. mc-cmd:: insecure :option: @@ -803,7 +833,7 @@ Syntax The full path to the bucket on which to remove the bucket replication configuration. Specify the :mc:`alias ` of a configured MinIO service as the prefix to the - :mc-cmd:`~mc replicate edit SOURCE` path. For example: + :mc-cmd:`~mc replicate rm SOURCE` path. For example: .. code-block:: shell @@ -827,4 +857,40 @@ Syntax *Optional* - Required if specifying :mc-cmd-option:`~mc replicate rm all` . \ No newline at end of file + Required if specifying :mc-cmd-option:`~mc replicate rm all` . + +.. mc-cmd:: resync, reset + :fullpath: + + Resynchronizes all objects in the specified bucket to the remote target + bucket. See :ref:`minio-replication-behavior-resync` for + more complete documentation. + + :mc-cmd:`mc replicate resync` has the following syntax: + + .. code-block:: shell + :class: copyable + + mc replicate resync SOURCE [args] + + :mc-cmd:`mc replicate resync` supports the following arguments: + + .. mc-cmd:: SOURCE + + *Required* + + The full path to the bucket on which to resync the bucket + replication status. Specify the + :mc:`alias ` of a configured MinIO service as the prefix to the + :mc-cmd:`~mc replicate edit SOURCE` path. For example: + + .. code-block:: shell + + mc replicate resync play/mybucket + + .. mc-cmd:: older-than + + *Optional* + + Specify a duration in days where MinIO only resynchronizes objects + older than the specified duration. diff --git a/source/replication/enable-server-side-one-way-bucket-replication.rst b/source/replication/enable-server-side-one-way-bucket-replication.rst index 1c8edd40..ffd7b09e 100644 --- a/source/replication/enable-server-side-one-way-bucket-replication.rst +++ b/source/replication/enable-server-side-one-way-bucket-replication.rst @@ -239,18 +239,17 @@ Considerations Replication of Existing Objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -MinIO performs replication as part of writing an object (PUT operations). MinIO -does *not* apply replication rules to existing objects in the bucket. Use -:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination -bucket. +Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z +` and :mc:`minio` +:minio-git:`RELEASE.2021-06-07T21-40-51Z +`, MinIO supports automatically +replicating existing objects in a bucket. -For buckets with active write operations during the procedure, any objects -written *before* configuring bucket replication remain unreplicated. - -Consider scheduling a maintenance period during which applications stop -all write operations to the bucket or buckets for which you are configuring -bucket replication. Restart write operations at the completion of the -procedure to ensure consistent object replication. +MinIO requires explicitly enabling replication of existing objects using the +:mc-cmd-option:`mc replicate add replicate` or +:mc-cmd-option:`mc replicate edit replicate` and including the +``existing-objects`` replication feature flag. This procedure includes the +required flags for enabling replication of existing objects. Replication of Delete Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -399,7 +398,7 @@ replication rule to the source MinIO cluster. mc replicate add AlphaReplication/SOURCEBUCKET \ --remote-bucket DESTINATIONBUCKET \ --arn 'arn:minio:replication:::DESTINATIONBUCKET' \ - --replicate "delete,delete-marker" + --replicate "delete,delete-marker,existing-objects" - Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha replicates data. The name *must* match the bucket specified when @@ -414,12 +413,15 @@ replication rule to the source MinIO cluster. :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the cluster. -- The ``--replicate "delete,delete-marker"`` flag enables replicating delete - markers and deletion of object versions. See - :mc-cmd-option:`mc replicate add replicate` for more complete - documentation. Omit these fields to disable replication of delete - operations. - +- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables + the following replication features: + + - :ref:`Replication of Deletes ` + - :ref:`Replication of existing Objects ` + + See :mc-cmd-option:`mc replicate add replicate` for more complete + documentation. Omit these fields to disable replication of delete operations + or replication of existing objects respectively. Specify any other supported optional arguments for :mc-cmd:`mc replicate add`. diff --git a/source/replication/enable-server-side-two-way-bucket-replication.rst b/source/replication/enable-server-side-two-way-bucket-replication.rst index 5513717c..67a97f26 100644 --- a/source/replication/enable-server-side-two-way-bucket-replication.rst +++ b/source/replication/enable-server-side-two-way-bucket-replication.rst @@ -238,18 +238,17 @@ Considerations Replication of Existing Objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -MinIO performs replication as part of writing an object (PUT operations). MinIO -does *not* apply replication rules to existing objects in the bucket. Use -:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination -bucket. +Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z +` and :mc:`minio` +:minio-git:`RELEASE.2021-06-07T21-40-51Z +`, MinIO supports automatically +replicating existing objects in a bucket. -For buckets with active write operations during the procedure, any objects -written *before* configuring bucket replication remain unreplicated. - -Consider scheduling a maintenance period during which applications stop -all write operations to the bucket or buckets for which you are configuring -bucket replication. Restart write operations at the completion of the -procedure to ensure consistent object replication. +MinIO requires explicitly enabling replication of existing objects using the +:mc-cmd-option:`mc replicate add replicate` or +:mc-cmd-option:`mc replicate edit replicate` and including the +``existing-objects`` replication feature flag. This procedure includes the +required flags for enabling replication of existing objects. Replication of Delete Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -460,7 +459,7 @@ A\) Create Replication Rule on Alpha mc replicate add AlphaReplication/SOURCEBUCKET \ --remote-bucket DESTINATIONBUCKET \ --arn 'arn:minio:replication:::DESTINATIONBUCKET' \ - --replicate "delete,delete-marker" + --replicate "delete,delete-marker,existing-objects" - Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha replicates data. The name *must* match the bucket specified when @@ -475,11 +474,15 @@ A\) Create Replication Rule on Alpha :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the cluster. - - The ``--replicate "delete,delete-marker"`` flag enables replicating delete - markers and deletion of object versions. See - :mc-cmd-option:`mc replicate add replicate` for more complete - documentation. Omit these fields to disable replication of delete - operations. + - The ``--replicate "delete,delete-marker,existing-objects"`` flag enables + the following replication features: + + - :ref:`Replication of Deletes ` + - :ref:`Replication of existing Objects ` + + See :mc-cmd-option:`mc replicate add replicate` for more complete + documentation. Omit these fields to disable replication of delete operations + or replication of existing objects respectively. Specify any other supported optional arguments for :mc-cmd:`mc replicate add`. @@ -496,7 +499,7 @@ B\) Create Replication Rule on Baker mc replicate add BakerReplication/SOURCEBUCKET \ --remote-bucket DESTINATIONBUCKET \ --arn 'arn:minio:replication:::DESTINATIONBUCKET' \ - --replicate "delete,delete-marker" + --replicate "delete,delete-marker,existing-objects" - Replace ``SOURCEBUCKET`` with the name of the bucket from which Baker replicates data. The name *must* match the bucket specified when @@ -511,11 +514,18 @@ B\) Create Replication Rule on Baker :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the cluster. - - The ``--replicate "delete,delete-marker"`` flag enables replicating delete - markers and deletion of object versions. See - :mc-cmd-option:`mc replicate add replicate` for more complete - documentation. Omit these fields to disable replication of delete - operations. + - The ``--replicate "delete,delete-marker,existing-objects"`` flag enables + the following replication features: + + - :ref:`Replication of Deletes ` + - :ref:`Replication of existing Objects ` + + See :mc-cmd-option:`mc replicate add replicate` for more complete + documentation. Omit these fields to disable replication of delete operations + or replication fof existing objects respectively. + + Specify any other supported optional arguments for + :mc-cmd:`mc replicate add`. Specify any other supported optional arguments for :mc-cmd:`mc replicate add`. diff --git a/source/replication/replication-overview.rst b/source/replication/replication-overview.rst index 110e1e11..ac08d763 100644 --- a/source/replication/replication-overview.rst +++ b/source/replication/replication-overview.rst @@ -134,8 +134,38 @@ You must explicitly enable synchronous replication when configuring the remote target target using the :mc-cmd:`mc admin bucket remote add` command with the :mc-cmd-option:`~mc admin bucket remote add sync` flag. +.. _minio-replication-behavior-resync: + +Resynchronization +~~~~~~~~~~~~~~~~~ + +The :mc-cmd:`mc replicate resync` command can reset replication synchronization +for a remote target. The resynchronization process attempts to queue all +objects on the source bucket regardless of their +:ref:`replication status `. Resynchronization +primarily supports recovery after partial or total loss of a remote +replication target. + +Initiating resynchronization on a bucket uses the same core +:ref:`replication scanner and queue system ` +for detecting and synchronizing objects. MinIO does not prioritize objects +in the target bucket, nor does it empty or otherwise modify the queue to +favor objects in the target bucket. MinIO skips synchronizing those +objects whose remote copy exactly match the source, including object +metadata. + +:mc-cmd:`mc replicate resync` operates at the bucket level and does +*not* support prefix-level granularity. Initiating resynchronization on a large +bucket may result in a significant increase in replication-related load +and traffic. Use this command with caution and only when necessary. + +Replication Behavior +~~~~~~~~~~~~~~~~~~~~ + +.. _minio-replication-behavior-delete: + Replication of Delete Operations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++ MinIO supports replicating delete operations, where MinIO synchronizes deleting specific object versions *and* new :s3-docs:`delete markers @@ -143,6 +173,10 @@ deleting specific object versions *and* new :s3-docs:`delete markers the same :ref:`replication process ` as all other replication operations. +MinIO only replicates explicit client-driven delete operations. MinIO does *not* +replicate objects deleted due to :ref:`lifecycle management expiration rules +`. + For delete marker replication, MinIO begins the replication process after a delete operation creates the delete marker. MinIO uses the ``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking @@ -166,6 +200,85 @@ specify both or either ``delete`` and ``delete-marker`` to enable versioned deletes and delete marker replication respectively. To enable both, specify both strings using a comma separator ``delete,delete-marker``. +.. admonition:: MinIO Trims Empty Object Prefixes on Source and Remote Bucket + :class: note, dropdown + + If a delete operation removes the last object in a bucket prefix, MinIO + recursively removes each empty part of the prefix up to the bucket root. + MinIO only applies the recursive removal to prefixes created *implicitly* as + part of object write operations - that is, the prefix was not created using + an explicit directory creation command such as :mc:`mc mb`. + + If a replication rule enables replication delete operations, the replication + process *also* applies the implicit prefix trimming behavior on the + destination MinIO cluster. + + For example, consider a bucket ``photos`` with the following object prefixes: + + - ``photos/2021/january/myphoto.jpg`` + - ``photos/2021/february/myotherphoto.jpg`` + - ``photos/NYE21/NewYears.jpg`` + + ``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`. + All other prefixes were *implicitly* created as part of writing the object + located at that prefix. + + - A command removes ``myphoto.jpg``. MinIO automatically trims the empty + ``/janaury`` prefix. + + - A command then removes the ``myotherphoto.jpg``. MinIO automatically + trims the ``/february`` prefix *and* the now-empty ``/2021`` prefix. + + - A command removes the ``NewYears.jpg`` object. MinIO leaves the + ``/NYE21`` prefix remains in place since it was *explicitly* created. + +.. _minio-replication-behavior-existing-objects: + +Replication of Existing Objects ++++++++++++++++++++++++++++++++ + +MinIO by default does not enable existing object replication. Objects +created before replication was configured *or* while replication is +disabled are not synchronized to the target deployment. +Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z +` and :mc:`minio` +:minio-git:`RELEASE.2021-06-07T21-40-51Z +`, MinIO supports enabling +replication of existing objects in a bucket. + +Enabling existing object replication marks all objects or object prefixes that +satisfy the replication rules as eligible for synchronization to the source +cluster, *even if* those objects were created prior to configuring or enabling +replication. You can enable existing object replication while configuring +or modifying a replication rule: + +- For new replication rules, include ``"existing-objects"`` to the list of + replication features specified to :mc-cmd-option:`mc replicate add replicate`. + +- For existing replication rules, add ``"existing-objects"`` to the list of + existing replication features using + :mc-cmd-option:`mc replicate edit replicate`. You must specify *all* desired + replication features when editing the replication rule. + +Enabling existing object replication does not increase the priority of objects +pending replication. MinIO uses the same core +:ref:`replication scanner and queue system ` for +detecting and synchronizing objects regardless of the enabled replication +feature. The time required to fully synchronize a bucket depends on a number of +factors, including but not limited to the current cluster replication load, +overall cluster load, and the size of the namespace (all objects in the bucket). + +MinIO does not synchronize existing unversioned objects. Specifically, the +bucket *must* have :ref:`versioning ` enabled when the +object was created. + +MinIO existing object replication +implements functionality similar to +`AWS: Replicating existing objects between S3 buckets +`__ +without the overhead of contacting technical support. + + .. toctree:: :hidden: :titlesonly: @@ -181,4 +294,3 @@ Client-Side Bucket Replication The :mc:`mc` command :mc-cmd:`mc mirror` supports watching a source bucket and automatically replicating objects to a destination bucket. -