mirror of
https://github.com/minio/docs.git
synced 2025-07-30 07:03:26 +03:00
DOCS-352: Add replica metadata sync, existing object replication
Apply suggestions from code review Co-authored-by: Poorna Krishnamoorthy <poornas@users.noreply.github.com>
This commit is contained in:
ravindk89
Ravind Kumar
@ -239,18 +239,17 @@ Considerations
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO performs replication as part of writing an object (PUT operations). MinIO
|
||||
does *not* apply replication rules to existing objects in the bucket. Use
|
||||
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
||||
bucket.
|
||||
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
|
||||
replicating existing objects in a bucket.
|
||||
|
||||
For buckets with active write operations during the procedure, any objects
|
||||
written *before* configuring bucket replication remain unreplicated.
|
||||
|
||||
Consider scheduling a maintenance period during which applications stop
|
||||
all write operations to the bucket or buckets for which you are configuring
|
||||
bucket replication. Restart write operations at the completion of the
|
||||
procedure to ensure consistent object replication.
|
||||
MinIO requires explicitly enabling replication of existing objects using the
|
||||
:mc-cmd-option:`mc replicate add replicate` or
|
||||
:mc-cmd-option:`mc replicate edit replicate` and including the
|
||||
``existing-objects`` replication feature flag. This procedure includes the
|
||||
required flags for enabling replication of existing objects.
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
@ -399,7 +398,7 @@ replication rule to the source MinIO cluster.
|
||||
mc replicate add AlphaReplication/SOURCEBUCKET \
|
||||
--remote-bucket DESTINATIONBUCKET \
|
||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||
--replicate "delete,delete-marker"
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
||||
replicates data. The name *must* match the bucket specified when
|
||||
@ -414,12 +413,15 @@ replication rule to the source MinIO cluster.
|
||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||
on the cluster.
|
||||
|
||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
||||
markers and deletion of object versions. See
|
||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete
|
||||
operations.
|
||||
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||
the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete operations
|
||||
or replication of existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for
|
||||
:mc-cmd:`mc replicate add`.
|
||||
|
||||
@ -238,18 +238,17 @@ Considerations
|
||||
Replication of Existing Objects
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
MinIO performs replication as part of writing an object (PUT operations). MinIO
|
||||
does *not* apply replication rules to existing objects in the bucket. Use
|
||||
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
||||
bucket.
|
||||
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
|
||||
replicating existing objects in a bucket.
|
||||
|
||||
For buckets with active write operations during the procedure, any objects
|
||||
written *before* configuring bucket replication remain unreplicated.
|
||||
|
||||
Consider scheduling a maintenance period during which applications stop
|
||||
all write operations to the bucket or buckets for which you are configuring
|
||||
bucket replication. Restart write operations at the completion of the
|
||||
procedure to ensure consistent object replication.
|
||||
MinIO requires explicitly enabling replication of existing objects using the
|
||||
:mc-cmd-option:`mc replicate add replicate` or
|
||||
:mc-cmd-option:`mc replicate edit replicate` and including the
|
||||
``existing-objects`` replication feature flag. This procedure includes the
|
||||
required flags for enabling replication of existing objects.
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
@ -460,7 +459,7 @@ A\) Create Replication Rule on Alpha
|
||||
mc replicate add AlphaReplication/SOURCEBUCKET \
|
||||
--remote-bucket DESTINATIONBUCKET \
|
||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||
--replicate "delete,delete-marker"
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
||||
replicates data. The name *must* match the bucket specified when
|
||||
@ -475,11 +474,15 @@ A\) Create Replication Rule on Alpha
|
||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||
on the cluster.
|
||||
|
||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
||||
markers and deletion of object versions. See
|
||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete
|
||||
operations.
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||
the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete operations
|
||||
or replication of existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for
|
||||
:mc-cmd:`mc replicate add`.
|
||||
@ -496,7 +499,7 @@ B\) Create Replication Rule on Baker
|
||||
mc replicate add BakerReplication/SOURCEBUCKET \
|
||||
--remote-bucket DESTINATIONBUCKET \
|
||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||
--replicate "delete,delete-marker"
|
||||
--replicate "delete,delete-marker,existing-objects"
|
||||
|
||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Baker
|
||||
replicates data. The name *must* match the bucket specified when
|
||||
@ -511,11 +514,18 @@ B\) Create Replication Rule on Baker
|
||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||
on the cluster.
|
||||
|
||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
||||
markers and deletion of object versions. See
|
||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete
|
||||
operations.
|
||||
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||
the following replication features:
|
||||
|
||||
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||
|
||||
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||
documentation. Omit these fields to disable replication of delete operations
|
||||
or replication fof existing objects respectively.
|
||||
|
||||
Specify any other supported optional arguments for
|
||||
:mc-cmd:`mc replicate add`.
|
||||
|
||||
Specify any other supported optional arguments for
|
||||
:mc-cmd:`mc replicate add`.
|
||||
|
||||
@ -134,8 +134,38 @@ You must explicitly enable synchronous replication when configuring the remote
|
||||
target target using the :mc-cmd:`mc admin bucket remote add` command with the
|
||||
:mc-cmd-option:`~mc admin bucket remote add sync` flag.
|
||||
|
||||
.. _minio-replication-behavior-resync:
|
||||
|
||||
Resynchronization
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The :mc-cmd:`mc replicate resync` command can reset replication synchronization
|
||||
for a remote target. The resynchronization process attempts to queue all
|
||||
objects on the source bucket regardless of their
|
||||
:ref:`replication status <minio-replication-process>`. Resynchronization
|
||||
primarily supports recovery after partial or total loss of a remote
|
||||
replication target.
|
||||
|
||||
Initiating resynchronization on a bucket uses the same core
|
||||
:ref:`replication scanner and queue system <minio-replication-process>`
|
||||
for detecting and synchronizing objects. MinIO does not prioritize objects
|
||||
in the target bucket, nor does it empty or otherwise modify the queue to
|
||||
favor objects in the target bucket. MinIO skips synchronizing those
|
||||
objects whose remote copy exactly match the source, including object
|
||||
metadata.
|
||||
|
||||
:mc-cmd:`mc replicate resync` operates at the bucket level and does
|
||||
*not* support prefix-level granularity. Initiating resynchronization on a large
|
||||
bucket may result in a significant increase in replication-related load
|
||||
and traffic. Use this command with caution and only when necessary.
|
||||
|
||||
Replication Behavior
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. _minio-replication-behavior-delete:
|
||||
|
||||
Replication of Delete Operations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
++++++++++++++++++++++++++++++++
|
||||
|
||||
MinIO supports replicating delete operations, where MinIO synchronizes
|
||||
deleting specific object versions *and* new :s3-docs:`delete markers
|
||||
@ -143,6 +173,10 @@ deleting specific object versions *and* new :s3-docs:`delete markers
|
||||
the same :ref:`replication process <minio-replication-process>` as all other
|
||||
replication operations.
|
||||
|
||||
MinIO only replicates explicit client-driven delete operations. MinIO does *not*
|
||||
replicate objects deleted due to :ref:`lifecycle management expiration rules
|
||||
<minio-lifecycle-management-expiration>`.
|
||||
|
||||
For delete marker replication, MinIO begins the replication process after
|
||||
a delete operation creates the delete marker. MinIO uses the
|
||||
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
|
||||
@ -166,6 +200,85 @@ specify both or either ``delete`` and ``delete-marker`` to enable versioned
|
||||
deletes and delete marker replication respectively. To enable both, specify both
|
||||
strings using a comma separator ``delete,delete-marker``.
|
||||
|
||||
.. admonition:: MinIO Trims Empty Object Prefixes on Source and Remote Bucket
|
||||
:class: note, dropdown
|
||||
|
||||
If a delete operation removes the last object in a bucket prefix, MinIO
|
||||
recursively removes each empty part of the prefix up to the bucket root.
|
||||
MinIO only applies the recursive removal to prefixes created *implicitly* as
|
||||
part of object write operations - that is, the prefix was not created using
|
||||
an explicit directory creation command such as :mc:`mc mb`.
|
||||
|
||||
If a replication rule enables replication delete operations, the replication
|
||||
process *also* applies the implicit prefix trimming behavior on the
|
||||
destination MinIO cluster.
|
||||
|
||||
For example, consider a bucket ``photos`` with the following object prefixes:
|
||||
|
||||
- ``photos/2021/january/myphoto.jpg``
|
||||
- ``photos/2021/february/myotherphoto.jpg``
|
||||
- ``photos/NYE21/NewYears.jpg``
|
||||
|
||||
``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`.
|
||||
All other prefixes were *implicitly* created as part of writing the object
|
||||
located at that prefix.
|
||||
|
||||
- A command removes ``myphoto.jpg``. MinIO automatically trims the empty
|
||||
``/janaury`` prefix.
|
||||
|
||||
- A command then removes the ``myotherphoto.jpg``. MinIO automatically
|
||||
trims the ``/february`` prefix *and* the now-empty ``/2021`` prefix.
|
||||
|
||||
- A command removes the ``NewYears.jpg`` object. MinIO leaves the
|
||||
``/NYE21`` prefix remains in place since it was *explicitly* created.
|
||||
|
||||
.. _minio-replication-behavior-existing-objects:
|
||||
|
||||
Replication of Existing Objects
|
||||
+++++++++++++++++++++++++++++++
|
||||
|
||||
MinIO by default does not enable existing object replication. Objects
|
||||
created before replication was configured *or* while replication is
|
||||
disabled are not synchronized to the target deployment.
|
||||
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports enabling
|
||||
replication of existing objects in a bucket.
|
||||
|
||||
Enabling existing object replication marks all objects or object prefixes that
|
||||
satisfy the replication rules as eligible for synchronization to the source
|
||||
cluster, *even if* those objects were created prior to configuring or enabling
|
||||
replication. You can enable existing object replication while configuring
|
||||
or modifying a replication rule:
|
||||
|
||||
- For new replication rules, include ``"existing-objects"`` to the list of
|
||||
replication features specified to :mc-cmd-option:`mc replicate add replicate`.
|
||||
|
||||
- For existing replication rules, add ``"existing-objects"`` to the list of
|
||||
existing replication features using
|
||||
:mc-cmd-option:`mc replicate edit replicate`. You must specify *all* desired
|
||||
replication features when editing the replication rule.
|
||||
|
||||
Enabling existing object replication does not increase the priority of objects
|
||||
pending replication. MinIO uses the same core
|
||||
:ref:`replication scanner and queue system <minio-replication-process>` for
|
||||
detecting and synchronizing objects regardless of the enabled replication
|
||||
feature. The time required to fully synchronize a bucket depends on a number of
|
||||
factors, including but not limited to the current cluster replication load,
|
||||
overall cluster load, and the size of the namespace (all objects in the bucket).
|
||||
|
||||
MinIO does not synchronize existing unversioned objects. Specifically, the
|
||||
bucket *must* have :ref:`versioning <minio-bucket-versioning>` enabled when the
|
||||
object was created.
|
||||
|
||||
MinIO existing object replication
|
||||
implements functionality similar to
|
||||
`AWS: Replicating existing objects between S3 buckets
|
||||
<https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/>`__
|
||||
without the overhead of contacting technical support.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:titlesonly:
|
||||
@ -181,4 +294,3 @@ Client-Side Bucket Replication
|
||||
|
||||
The :mc:`mc` command :mc-cmd:`mc mirror` supports watching a source bucket
|
||||
and automatically replicating objects to a destination bucket.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user