mirror of
https://github.com/minio/docs.git
synced 2025-07-30 07:03:26 +03:00
DOCS-352: Add replica metadata sync, existing object replication
Apply suggestions from code review Co-authored-by: Poorna Krishnamoorthy <poornas@users.noreply.github.com>
This commit is contained in:
@ -92,6 +92,10 @@ options to create new expiration rules on a bucket:
|
|||||||
- :mc-cmd-option:`mc ilm add expiry-days` to expire objects after a
|
- :mc-cmd-option:`mc ilm add expiry-days` to expire objects after a
|
||||||
specified number of calendar days.
|
specified number of calendar days.
|
||||||
|
|
||||||
|
For buckets with :ref:`replication <minio-bucket-replication>` configured, MinIO
|
||||||
|
does not replicate objects deleted by a lifecycle management expiration rule.
|
||||||
|
See :ref:`minio-replication-behavior-delete` for more information.
|
||||||
|
|
||||||
Versioned Buckets
|
Versioned Buckets
|
||||||
~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
@ -165,15 +165,49 @@ documentation on adding users and policies to a MinIO cluster.
|
|||||||
Replication of Existing Objects
|
Replication of Existing Objects
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
MinIO performs replication as part of writing an object (PUT operations).
|
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||||
MinIO does *not* apply replication rules to existing objects written
|
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||||
*before* enabling replication.
|
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||||
|
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
|
||||||
|
replicating existing objects in a bucket. MinIO existing object replication
|
||||||
|
implements functionality similar to
|
||||||
|
`AWS: Replicating existing objects between S3 buckets
|
||||||
|
<https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/>`__
|
||||||
|
without the overhead of contacting technical support.
|
||||||
|
|
||||||
For buckets with existing objects, consider using :mc:`mc mirror` or
|
- To enable replication of existing objects when creating a new replication
|
||||||
:mc:`mc cp` to seed the destination bucket. Consider scheduling a maintenance
|
rule, include ``"existing-objects"`` to the list of replication features
|
||||||
window during which applications stop writes to the bucket. Once
|
specified to :mc-cmd-option:`mc replicate add replicate`.
|
||||||
the :mc:`~mc mirror` or :mc:`~mc cp` fully sync the source and destination,
|
|
||||||
enable bucket replication and resume normal operations on the bucket.
|
- To enable replication of existing objects for an existing replication rule,
|
||||||
|
add ``"existing-objects"`` to the list of existing replication features using
|
||||||
|
:mc-cmd-option:`mc replicate add replicate`. You must specify *all*
|
||||||
|
desired replication features when editing the replication rule.
|
||||||
|
|
||||||
|
See :ref:`minio-replication-behavior-existing-objects` for more complete
|
||||||
|
documentation on this behavior.
|
||||||
|
|
||||||
|
Synchronization of Metadata Changes
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
MinIO supports :ref:`two-way active-active
|
||||||
|
<minio-bucket-replication-serverside-twoway>` replication configurations, where
|
||||||
|
MinIO synchronizes new and modified objects between a bucket on two MinIO
|
||||||
|
deployments. Starting with :mc:`mc` :minio-git:`RELEASE.2021-05-18T03-39-44Z
|
||||||
|
<mc/release/tag/RELEASE.2021-05-18T03-39-44Z>`, MinIO by default synchronizes
|
||||||
|
metadata-only changes to a replicated object back to the "source" deployment.
|
||||||
|
Prior to the this update, MinIO did not support synchronizing metadata-only
|
||||||
|
changes to a replicated object.
|
||||||
|
|
||||||
|
With metadata synchronization enabled, MinIO resets the object
|
||||||
|
:ref:`replication status <minio-replication-process>` to indicate
|
||||||
|
replication eligibility. Specifically, when an application performs a
|
||||||
|
metadata-only update to an object with the ``REPLICA`` status, MinIO marks the
|
||||||
|
object as ``PENDING`` and eligible for replication.
|
||||||
|
|
||||||
|
To disable metadata synchronization, use the
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate` command and omit
|
||||||
|
``replica-metadata-sync`` from the replication feature list.
|
||||||
|
|
||||||
Replication of Delete Operations
|
Replication of Delete Operations
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
@ -189,47 +223,16 @@ of specific versioned objects:
|
|||||||
- For delete operations on versions of an object,
|
- For delete operations on versions of an object,
|
||||||
MinIO replication also deletes those versions on the target bucket.
|
MinIO replication also deletes those versions on the target bucket.
|
||||||
|
|
||||||
|
MinIO does *not* replicate objects deleted due to
|
||||||
|
:ref:`lifecycle management expiration rules
|
||||||
|
<minio-lifecycle-management-expiration>`. MinIO only replicates explicit
|
||||||
|
client-driven delete operations.
|
||||||
|
|
||||||
MinIO requires explicitly enabling replication of delete operations using the
|
MinIO requires explicitly enabling replication of delete operations using the
|
||||||
:mc-cmd-option:`mc replicate add replicate` flag. This procedure includes the
|
:mc-cmd-option:`mc replicate add replicate` flag. This procedure includes the
|
||||||
required flags for enabling replication of delete operations and delete markers.
|
required flags for enabling replication of delete operations and delete markers.
|
||||||
|
See :ref:`minio-replication-behavior-delete` for more complete documentation
|
||||||
.. note::
|
on this behavior.
|
||||||
|
|
||||||
If a delete operation removes the last object in a bucket prefix, MinIO
|
|
||||||
recursively deletes any empty directories within that prefix. For example:
|
|
||||||
|
|
||||||
``play/mybucket/path/to/foo.txt``
|
|
||||||
|
|
||||||
The bucket prefix is ``/path/to/foo.txt``. If ``foo.txt`` is the last object
|
|
||||||
in prefix, MinIO deletes the entire prefix. If an object exists at ``/path``,
|
|
||||||
MinIO stops deleting the prefix at that point.
|
|
||||||
|
|
||||||
.. admonition:: MinIO Trims Empty Object Prefixes
|
|
||||||
|
|
||||||
If a delete operation removes the last object in a bucket prefix, MinIO
|
|
||||||
recursively removes each empty part of the prefix up to the bucket root.
|
|
||||||
MinIO only applies the recursive removal to prefixes created *implicitly* as
|
|
||||||
part of object write operations - that is, the prefix was not created using
|
|
||||||
an explicit directory creation command such as :mc:`mc mb`.
|
|
||||||
|
|
||||||
If a replication rule enables replication delete operations, the replication
|
|
||||||
process *also* applies the implicit prefix trimming behavior on the
|
|
||||||
destination MinIO cluster.
|
|
||||||
|
|
||||||
For example, consider a bucket ``photos`` with the following object prefixes:
|
|
||||||
|
|
||||||
- ``photos/2021/january/myphoto.jpg``
|
|
||||||
- ``photos/2021/february/myotherphoto.jpg``
|
|
||||||
- ``photos/NYE21/NewYears.jpg``
|
|
||||||
|
|
||||||
``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`.
|
|
||||||
All other prefixes were *implicitly* created as part of writing the object
|
|
||||||
located at that prefix. If a command removes ``myphoto.jpg``, it also
|
|
||||||
automatically trims the empty ``/janaury`` prefix. If <command> then removes
|
|
||||||
the ``myotherphoto.jpg``, it also automatically trims both the ``/february``
|
|
||||||
prefix *and* the now-empty ``/2021`` prefix. If <command> removes the
|
|
||||||
``NewYears.jpg`` object, the ``/NYE21`` prefix remains in place since it was
|
|
||||||
*explicitly* created.
|
|
||||||
|
|
||||||
Replication of Encrypted Objects
|
Replication of Encrypted Objects
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
@ -448,7 +451,7 @@ Syntax
|
|||||||
*Optional*
|
*Optional*
|
||||||
|
|
||||||
Specify a comma-separated list of the following values to enable extended
|
Specify a comma-separated list of the following values to enable extended
|
||||||
replication features:
|
replication features.
|
||||||
|
|
||||||
- ``delete`` - Directs MinIO to replicate DELETE operations to the
|
- ``delete`` - Directs MinIO to replicate DELETE operations to the
|
||||||
destination bucket.
|
destination bucket.
|
||||||
@ -456,6 +459,8 @@ Syntax
|
|||||||
- ``delete-marker`` - Directs MinIO to replicate delete markers to the
|
- ``delete-marker`` - Directs MinIO to replicate delete markers to the
|
||||||
destination bucket.
|
destination bucket.
|
||||||
|
|
||||||
|
- ``existing-objects`` - Directs MinIO to replicate objects created
|
||||||
|
before replication was enabled *or* while replication was suspended.
|
||||||
|
|
||||||
.. mc-cmd:: tags
|
.. mc-cmd:: tags
|
||||||
:option:
|
:option:
|
||||||
@ -522,6 +527,15 @@ Syntax
|
|||||||
does not begin replicating objects using the rule until it
|
does not begin replicating objects using the rule until it
|
||||||
is enabled using :mc-cmd:`mc replicate edit`.
|
is enabled using :mc-cmd:`mc replicate edit`.
|
||||||
|
|
||||||
|
Objects created while replication is disabled are not
|
||||||
|
immediately eligible for replication after enabling the rule.
|
||||||
|
You must explicitly enable replication of existing
|
||||||
|
objects by including ``"existing-objects"`` to the list of
|
||||||
|
replication features specified to
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate`. See
|
||||||
|
:ref:`minio-replication-behavior-existing-objects` for more
|
||||||
|
information.
|
||||||
|
|
||||||
|
|
||||||
.. mc-cmd:: edit
|
.. mc-cmd:: edit
|
||||||
:fullpath:
|
:fullpath:
|
||||||
@ -581,10 +595,21 @@ Syntax
|
|||||||
- ``delete-marker`` - Directs MinIO to replicate delete markers to the
|
- ``delete-marker`` - Directs MinIO to replicate delete markers to the
|
||||||
destination bucket.
|
destination bucket.
|
||||||
|
|
||||||
MinIO does *not* apply the updated replication rules to existing
|
- ``replica-metadata-sync`` - Directs MinIO to synchronize metadata-only
|
||||||
objects in the source bucket. For example, enabling delete marker
|
changes on a replicated object back to the source. This feature only
|
||||||
replication does not automatically replicate existing objects with
|
effects :ref:`two-way active-active
|
||||||
delete markers.
|
<minio-bucket-replication-serverside-twoway>` replication
|
||||||
|
configurations.
|
||||||
|
|
||||||
|
Omitting this value directs MinIO to stop replicating metadata-only
|
||||||
|
changes back to the source.
|
||||||
|
|
||||||
|
- ``existing-objects`` - Directs MinIO to replicate objects created
|
||||||
|
prior to configuring or enabling replication. MinIO by default does
|
||||||
|
*not* synchronize existing objects to the remote target.
|
||||||
|
|
||||||
|
See :ref:`minio-replication-behavior-existing-objects` for more
|
||||||
|
information.
|
||||||
|
|
||||||
|
|
||||||
.. mc-cmd:: tags
|
.. mc-cmd:: tags
|
||||||
@ -639,12 +664,17 @@ Syntax
|
|||||||
Enables or disables the replication rule. Specify one of the following
|
Enables or disables the replication rule. Specify one of the following
|
||||||
values:
|
values:
|
||||||
|
|
||||||
- ``"enable"`` - Enables the replication rule. MinIO begins replicating
|
- ``"enable"`` - Enables the replication rule.
|
||||||
objects written *after* enabling the rule. Existing objects require
|
|
||||||
manual migration to the destination bucket.
|
|
||||||
|
|
||||||
- ``"disable"`` - Disables the replication rule.
|
- ``"disable"`` - Disables the replication rule.
|
||||||
|
|
||||||
|
Objects created while replication is disabled are not immediately eligible
|
||||||
|
for replication after enabling the rule. You must explicitly enable
|
||||||
|
replication of existing objects by including ``"existing-objects"`` to the
|
||||||
|
list of replication features specified to
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate`. See
|
||||||
|
:ref:`minio-replication-behavior-existing-objects` for more information.
|
||||||
|
|
||||||
.. mc-cmd:: ls
|
.. mc-cmd:: ls
|
||||||
:fullpath:
|
:fullpath:
|
||||||
|
|
||||||
@ -666,11 +696,11 @@ Syntax
|
|||||||
The full path to the bucket on which to list the
|
The full path to the bucket on which to list the
|
||||||
replication configurations. Specify the
|
replication configurations. Specify the
|
||||||
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
||||||
:mc-cmd:`~mc replicate add SOURCE` path. For example:
|
:mc-cmd:`~mc replicate ls SOURCE` path. For example:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
|
|
||||||
mc replicate add play/mybucket
|
mc replicate ls play/mybucket
|
||||||
|
|
||||||
.. mc-cmd:: insecure
|
.. mc-cmd:: insecure
|
||||||
:option:
|
:option:
|
||||||
@ -765,11 +795,11 @@ Syntax
|
|||||||
The full path to the bucket to which to import the
|
The full path to the bucket to which to import the
|
||||||
replication configurations. Specify the
|
replication configurations. Specify the
|
||||||
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
||||||
:mc-cmd:`~mc replicate add SOURCE` path. For example:
|
:mc-cmd:`~mc replicate import SOURCE` path. For example:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
|
|
||||||
mc replicate export play/mybucket
|
mc replicate import play/mybucket
|
||||||
|
|
||||||
.. mc-cmd:: insecure
|
.. mc-cmd:: insecure
|
||||||
:option:
|
:option:
|
||||||
@ -803,7 +833,7 @@ Syntax
|
|||||||
The full path to the bucket on which to remove the bucket
|
The full path to the bucket on which to remove the bucket
|
||||||
replication configuration. Specify the
|
replication configuration. Specify the
|
||||||
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
||||||
:mc-cmd:`~mc replicate edit SOURCE` path. For example:
|
:mc-cmd:`~mc replicate rm SOURCE` path. For example:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
|
|
||||||
@ -828,3 +858,39 @@ Syntax
|
|||||||
*Optional*
|
*Optional*
|
||||||
|
|
||||||
Required if specifying :mc-cmd-option:`~mc replicate rm all` .
|
Required if specifying :mc-cmd-option:`~mc replicate rm all` .
|
||||||
|
|
||||||
|
.. mc-cmd:: resync, reset
|
||||||
|
:fullpath:
|
||||||
|
|
||||||
|
Resynchronizes all objects in the specified bucket to the remote target
|
||||||
|
bucket. See :ref:`minio-replication-behavior-resync` for
|
||||||
|
more complete documentation.
|
||||||
|
|
||||||
|
:mc-cmd:`mc replicate resync` has the following syntax:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
mc replicate resync SOURCE [args]
|
||||||
|
|
||||||
|
:mc-cmd:`mc replicate resync` supports the following arguments:
|
||||||
|
|
||||||
|
.. mc-cmd:: SOURCE
|
||||||
|
|
||||||
|
*Required*
|
||||||
|
|
||||||
|
The full path to the bucket on which to resync the bucket
|
||||||
|
replication status. Specify the
|
||||||
|
:mc:`alias <mc alias>` of a configured MinIO service as the prefix to the
|
||||||
|
:mc-cmd:`~mc replicate edit SOURCE` path. For example:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
mc replicate resync play/mybucket
|
||||||
|
|
||||||
|
.. mc-cmd:: older-than
|
||||||
|
|
||||||
|
*Optional*
|
||||||
|
|
||||||
|
Specify a duration in days where MinIO only resynchronizes objects
|
||||||
|
older than the specified duration.
|
||||||
|
@ -239,18 +239,17 @@ Considerations
|
|||||||
Replication of Existing Objects
|
Replication of Existing Objects
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
MinIO performs replication as part of writing an object (PUT operations). MinIO
|
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||||
does *not* apply replication rules to existing objects in the bucket. Use
|
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||||
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||||
bucket.
|
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
|
||||||
|
replicating existing objects in a bucket.
|
||||||
|
|
||||||
For buckets with active write operations during the procedure, any objects
|
MinIO requires explicitly enabling replication of existing objects using the
|
||||||
written *before* configuring bucket replication remain unreplicated.
|
:mc-cmd-option:`mc replicate add replicate` or
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate` and including the
|
||||||
Consider scheduling a maintenance period during which applications stop
|
``existing-objects`` replication feature flag. This procedure includes the
|
||||||
all write operations to the bucket or buckets for which you are configuring
|
required flags for enabling replication of existing objects.
|
||||||
bucket replication. Restart write operations at the completion of the
|
|
||||||
procedure to ensure consistent object replication.
|
|
||||||
|
|
||||||
Replication of Delete Operations
|
Replication of Delete Operations
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
@ -399,7 +398,7 @@ replication rule to the source MinIO cluster.
|
|||||||
mc replicate add AlphaReplication/SOURCEBUCKET \
|
mc replicate add AlphaReplication/SOURCEBUCKET \
|
||||||
--remote-bucket DESTINATIONBUCKET \
|
--remote-bucket DESTINATIONBUCKET \
|
||||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||||
--replicate "delete,delete-marker"
|
--replicate "delete,delete-marker,existing-objects"
|
||||||
|
|
||||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
||||||
replicates data. The name *must* match the bucket specified when
|
replicates data. The name *must* match the bucket specified when
|
||||||
@ -414,12 +413,15 @@ replication rule to the source MinIO cluster.
|
|||||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||||
on the cluster.
|
on the cluster.
|
||||||
|
|
||||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||||
markers and deletion of object versions. See
|
the following replication features:
|
||||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
|
||||||
documentation. Omit these fields to disable replication of delete
|
|
||||||
operations.
|
|
||||||
|
|
||||||
|
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||||
|
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||||
|
|
||||||
|
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||||
|
documentation. Omit these fields to disable replication of delete operations
|
||||||
|
or replication of existing objects respectively.
|
||||||
|
|
||||||
Specify any other supported optional arguments for
|
Specify any other supported optional arguments for
|
||||||
:mc-cmd:`mc replicate add`.
|
:mc-cmd:`mc replicate add`.
|
||||||
|
@ -238,18 +238,17 @@ Considerations
|
|||||||
Replication of Existing Objects
|
Replication of Existing Objects
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
MinIO performs replication as part of writing an object (PUT operations). MinIO
|
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||||
does *not* apply replication rules to existing objects in the bucket. Use
|
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||||
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||||
bucket.
|
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
|
||||||
|
replicating existing objects in a bucket.
|
||||||
|
|
||||||
For buckets with active write operations during the procedure, any objects
|
MinIO requires explicitly enabling replication of existing objects using the
|
||||||
written *before* configuring bucket replication remain unreplicated.
|
:mc-cmd-option:`mc replicate add replicate` or
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate` and including the
|
||||||
Consider scheduling a maintenance period during which applications stop
|
``existing-objects`` replication feature flag. This procedure includes the
|
||||||
all write operations to the bucket or buckets for which you are configuring
|
required flags for enabling replication of existing objects.
|
||||||
bucket replication. Restart write operations at the completion of the
|
|
||||||
procedure to ensure consistent object replication.
|
|
||||||
|
|
||||||
Replication of Delete Operations
|
Replication of Delete Operations
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
@ -460,7 +459,7 @@ A\) Create Replication Rule on Alpha
|
|||||||
mc replicate add AlphaReplication/SOURCEBUCKET \
|
mc replicate add AlphaReplication/SOURCEBUCKET \
|
||||||
--remote-bucket DESTINATIONBUCKET \
|
--remote-bucket DESTINATIONBUCKET \
|
||||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||||
--replicate "delete,delete-marker"
|
--replicate "delete,delete-marker,existing-objects"
|
||||||
|
|
||||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
||||||
replicates data. The name *must* match the bucket specified when
|
replicates data. The name *must* match the bucket specified when
|
||||||
@ -475,11 +474,15 @@ A\) Create Replication Rule on Alpha
|
|||||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||||
on the cluster.
|
on the cluster.
|
||||||
|
|
||||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||||
markers and deletion of object versions. See
|
the following replication features:
|
||||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
|
||||||
documentation. Omit these fields to disable replication of delete
|
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||||
operations.
|
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||||
|
|
||||||
|
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||||
|
documentation. Omit these fields to disable replication of delete operations
|
||||||
|
or replication of existing objects respectively.
|
||||||
|
|
||||||
Specify any other supported optional arguments for
|
Specify any other supported optional arguments for
|
||||||
:mc-cmd:`mc replicate add`.
|
:mc-cmd:`mc replicate add`.
|
||||||
@ -496,7 +499,7 @@ B\) Create Replication Rule on Baker
|
|||||||
mc replicate add BakerReplication/SOURCEBUCKET \
|
mc replicate add BakerReplication/SOURCEBUCKET \
|
||||||
--remote-bucket DESTINATIONBUCKET \
|
--remote-bucket DESTINATIONBUCKET \
|
||||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||||
--replicate "delete,delete-marker"
|
--replicate "delete,delete-marker,existing-objects"
|
||||||
|
|
||||||
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Baker
|
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Baker
|
||||||
replicates data. The name *must* match the bucket specified when
|
replicates data. The name *must* match the bucket specified when
|
||||||
@ -511,11 +514,18 @@ B\) Create Replication Rule on Baker
|
|||||||
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||||
on the cluster.
|
on the cluster.
|
||||||
|
|
||||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables
|
||||||
markers and deletion of object versions. See
|
the following replication features:
|
||||||
:mc-cmd-option:`mc replicate add replicate` for more complete
|
|
||||||
documentation. Omit these fields to disable replication of delete
|
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
|
||||||
operations.
|
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
|
||||||
|
|
||||||
|
See :mc-cmd-option:`mc replicate add replicate` for more complete
|
||||||
|
documentation. Omit these fields to disable replication of delete operations
|
||||||
|
or replication fof existing objects respectively.
|
||||||
|
|
||||||
|
Specify any other supported optional arguments for
|
||||||
|
:mc-cmd:`mc replicate add`.
|
||||||
|
|
||||||
Specify any other supported optional arguments for
|
Specify any other supported optional arguments for
|
||||||
:mc-cmd:`mc replicate add`.
|
:mc-cmd:`mc replicate add`.
|
||||||
|
@ -134,8 +134,38 @@ You must explicitly enable synchronous replication when configuring the remote
|
|||||||
target target using the :mc-cmd:`mc admin bucket remote add` command with the
|
target target using the :mc-cmd:`mc admin bucket remote add` command with the
|
||||||
:mc-cmd-option:`~mc admin bucket remote add sync` flag.
|
:mc-cmd-option:`~mc admin bucket remote add sync` flag.
|
||||||
|
|
||||||
|
.. _minio-replication-behavior-resync:
|
||||||
|
|
||||||
|
Resynchronization
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The :mc-cmd:`mc replicate resync` command can reset replication synchronization
|
||||||
|
for a remote target. The resynchronization process attempts to queue all
|
||||||
|
objects on the source bucket regardless of their
|
||||||
|
:ref:`replication status <minio-replication-process>`. Resynchronization
|
||||||
|
primarily supports recovery after partial or total loss of a remote
|
||||||
|
replication target.
|
||||||
|
|
||||||
|
Initiating resynchronization on a bucket uses the same core
|
||||||
|
:ref:`replication scanner and queue system <minio-replication-process>`
|
||||||
|
for detecting and synchronizing objects. MinIO does not prioritize objects
|
||||||
|
in the target bucket, nor does it empty or otherwise modify the queue to
|
||||||
|
favor objects in the target bucket. MinIO skips synchronizing those
|
||||||
|
objects whose remote copy exactly match the source, including object
|
||||||
|
metadata.
|
||||||
|
|
||||||
|
:mc-cmd:`mc replicate resync` operates at the bucket level and does
|
||||||
|
*not* support prefix-level granularity. Initiating resynchronization on a large
|
||||||
|
bucket may result in a significant increase in replication-related load
|
||||||
|
and traffic. Use this command with caution and only when necessary.
|
||||||
|
|
||||||
|
Replication Behavior
|
||||||
|
~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. _minio-replication-behavior-delete:
|
||||||
|
|
||||||
Replication of Delete Operations
|
Replication of Delete Operations
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
MinIO supports replicating delete operations, where MinIO synchronizes
|
MinIO supports replicating delete operations, where MinIO synchronizes
|
||||||
deleting specific object versions *and* new :s3-docs:`delete markers
|
deleting specific object versions *and* new :s3-docs:`delete markers
|
||||||
@ -143,6 +173,10 @@ deleting specific object versions *and* new :s3-docs:`delete markers
|
|||||||
the same :ref:`replication process <minio-replication-process>` as all other
|
the same :ref:`replication process <minio-replication-process>` as all other
|
||||||
replication operations.
|
replication operations.
|
||||||
|
|
||||||
|
MinIO only replicates explicit client-driven delete operations. MinIO does *not*
|
||||||
|
replicate objects deleted due to :ref:`lifecycle management expiration rules
|
||||||
|
<minio-lifecycle-management-expiration>`.
|
||||||
|
|
||||||
For delete marker replication, MinIO begins the replication process after
|
For delete marker replication, MinIO begins the replication process after
|
||||||
a delete operation creates the delete marker. MinIO uses the
|
a delete operation creates the delete marker. MinIO uses the
|
||||||
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
|
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
|
||||||
@ -166,6 +200,85 @@ specify both or either ``delete`` and ``delete-marker`` to enable versioned
|
|||||||
deletes and delete marker replication respectively. To enable both, specify both
|
deletes and delete marker replication respectively. To enable both, specify both
|
||||||
strings using a comma separator ``delete,delete-marker``.
|
strings using a comma separator ``delete,delete-marker``.
|
||||||
|
|
||||||
|
.. admonition:: MinIO Trims Empty Object Prefixes on Source and Remote Bucket
|
||||||
|
:class: note, dropdown
|
||||||
|
|
||||||
|
If a delete operation removes the last object in a bucket prefix, MinIO
|
||||||
|
recursively removes each empty part of the prefix up to the bucket root.
|
||||||
|
MinIO only applies the recursive removal to prefixes created *implicitly* as
|
||||||
|
part of object write operations - that is, the prefix was not created using
|
||||||
|
an explicit directory creation command such as :mc:`mc mb`.
|
||||||
|
|
||||||
|
If a replication rule enables replication delete operations, the replication
|
||||||
|
process *also* applies the implicit prefix trimming behavior on the
|
||||||
|
destination MinIO cluster.
|
||||||
|
|
||||||
|
For example, consider a bucket ``photos`` with the following object prefixes:
|
||||||
|
|
||||||
|
- ``photos/2021/january/myphoto.jpg``
|
||||||
|
- ``photos/2021/february/myotherphoto.jpg``
|
||||||
|
- ``photos/NYE21/NewYears.jpg``
|
||||||
|
|
||||||
|
``photos/NYE21`` is the *only* prefix explicitly created using :mc:`mc mb`.
|
||||||
|
All other prefixes were *implicitly* created as part of writing the object
|
||||||
|
located at that prefix.
|
||||||
|
|
||||||
|
- A command removes ``myphoto.jpg``. MinIO automatically trims the empty
|
||||||
|
``/janaury`` prefix.
|
||||||
|
|
||||||
|
- A command then removes the ``myotherphoto.jpg``. MinIO automatically
|
||||||
|
trims the ``/february`` prefix *and* the now-empty ``/2021`` prefix.
|
||||||
|
|
||||||
|
- A command removes the ``NewYears.jpg`` object. MinIO leaves the
|
||||||
|
``/NYE21`` prefix remains in place since it was *explicitly* created.
|
||||||
|
|
||||||
|
.. _minio-replication-behavior-existing-objects:
|
||||||
|
|
||||||
|
Replication of Existing Objects
|
||||||
|
+++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
|
MinIO by default does not enable existing object replication. Objects
|
||||||
|
created before replication was configured *or* while replication is
|
||||||
|
disabled are not synchronized to the target deployment.
|
||||||
|
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
|
||||||
|
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
|
||||||
|
:minio-git:`RELEASE.2021-06-07T21-40-51Z
|
||||||
|
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports enabling
|
||||||
|
replication of existing objects in a bucket.
|
||||||
|
|
||||||
|
Enabling existing object replication marks all objects or object prefixes that
|
||||||
|
satisfy the replication rules as eligible for synchronization to the source
|
||||||
|
cluster, *even if* those objects were created prior to configuring or enabling
|
||||||
|
replication. You can enable existing object replication while configuring
|
||||||
|
or modifying a replication rule:
|
||||||
|
|
||||||
|
- For new replication rules, include ``"existing-objects"`` to the list of
|
||||||
|
replication features specified to :mc-cmd-option:`mc replicate add replicate`.
|
||||||
|
|
||||||
|
- For existing replication rules, add ``"existing-objects"`` to the list of
|
||||||
|
existing replication features using
|
||||||
|
:mc-cmd-option:`mc replicate edit replicate`. You must specify *all* desired
|
||||||
|
replication features when editing the replication rule.
|
||||||
|
|
||||||
|
Enabling existing object replication does not increase the priority of objects
|
||||||
|
pending replication. MinIO uses the same core
|
||||||
|
:ref:`replication scanner and queue system <minio-replication-process>` for
|
||||||
|
detecting and synchronizing objects regardless of the enabled replication
|
||||||
|
feature. The time required to fully synchronize a bucket depends on a number of
|
||||||
|
factors, including but not limited to the current cluster replication load,
|
||||||
|
overall cluster load, and the size of the namespace (all objects in the bucket).
|
||||||
|
|
||||||
|
MinIO does not synchronize existing unversioned objects. Specifically, the
|
||||||
|
bucket *must* have :ref:`versioning <minio-bucket-versioning>` enabled when the
|
||||||
|
object was created.
|
||||||
|
|
||||||
|
MinIO existing object replication
|
||||||
|
implements functionality similar to
|
||||||
|
`AWS: Replicating existing objects between S3 buckets
|
||||||
|
<https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/>`__
|
||||||
|
without the overhead of contacting technical support.
|
||||||
|
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:hidden:
|
:hidden:
|
||||||
:titlesonly:
|
:titlesonly:
|
||||||
@ -181,4 +294,3 @@ Client-Side Bucket Replication
|
|||||||
|
|
||||||
The :mc:`mc` command :mc-cmd:`mc mirror` supports watching a source bucket
|
The :mc:`mc` command :mc-cmd:`mc mirror` supports watching a source bucket
|
||||||
and automatically replicating objects to a destination bucket.
|
and automatically replicating objects to a destination bucket.
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user