minio/docs
1
0
Fork 0
mirror of https://github.com/minio/docs.git synced 2025-07-05 03:21:23 +03:00
Code Activity

DOCS-412: Add replication resynchronization procedure

Browse Source 
Includes refinements to other replication procedures and single sourcing
This commit is contained in:
ravindk89
2022-01-05 17:37:44 -05:00
committed by Ravind Kumar
parent 7a99642e34
commit d92f9d4a2e
10 changed files with 2045 additions and 1426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1240
source/images/replication/active-active-twoway-replication.svg
View File

File diff suppressed because it is too large Load Diff
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 57 KiB

1593
source/images/replication/active-passive-oneway-replication.svg
View File

File diff suppressed because it is too large Load Diff
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 60 KiB

3
source/includes/common-minio-mc.rst
View File

@ -87,8 +87,7 @@ commands *may* work as documented, any such usage is at your own risk.
- Brackets ``[]`` indicate optional parameters.
- Parameters sharing a line are mutually dependent.
- Parameters sharing a line *and* seperated using the pipe ``|`` operator are
mutually exclusive.
- Parameters seperated using the pipe ``|`` operator are mutually exclusive.
Copy the example to a text editor and modify as-needed before running the
command in the terminal/shell.

39
source/includes/common-replication.rst Normal file
View File

@ -0,0 +1,39 @@
.. start-replication-encrypted-objects
MinIO supports replication of objects encrypted using :ref:`SSE-KMS <minio-encryption-sse-kms>` and :ref:`SSE-S3 <minio-encryption-sse-s3>`:
- For objects encrypted using SSE-KMS, MinIO *requires* that the target bucket support SSE-KMS encryption of objects using the *same key names* used to encrypt objects on the source bucket.
- For objects encrypted using :ref:`SSE-S3 <minio-encryption-sse-s3>`, MinIO *requires* that the target bucket also support SSE-S3 encryption of objects regardless of key name.
As part of the replication process, MinIO *decrypts* the object on the source bucket and transmits the unencrypted object over the network. The destination MinIO cluster then re-encrypts the object using the encryption settings from the source. MinIO therefore *strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and destination deployments to ensure the safety of objects during transmission.
MinIO does *not* support replicating client-side encrypted objects (SSE-C).
.. end-replication-encrypted-objects
.. start-replication-minio-only
MinIO server-side replication only works between MinIO clusters. Both the
source and destination clusters *must* run MinIO.
To configure replication between arbitrary S3-compatible services,
use :mc-cmd:`mc mirror`.
.. end-replication-minio-only
.. start-replication-requires-versioning
MinIO relies on the immutability protections provided by :ref:`versioning <minio-bucket-versioning>` to support replication and resynchronization.
Use :mc-cmd:`mc version info` to validate the versioning status of both the healthy source and unhealthy target buckets. Use the :mc-cmd:`mc version enable` command to enable versioning as necessary.
.. end-replication-requires-versioning
.. start-replication-requires-object-locking
MinIO supports replicating objects held under :ref:`WORM Locking <minio-object-locking>`. Both replication buckets *must* have object locking enabled for MinIO to replicate the locked object. For active-active configuration, MinIO recommends using the *same* retention rules on both buckets to ensure consistent behavior across sites.
You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket *prior* to beginning this procedure.
.. end-replication-requires-object-locking

48
source/reference/minio-mc/mc-replicate-resync.rst
View File

@ -46,7 +46,7 @@ for more information on active-active replication:
.. code-block:: shell
:class: copyable
mc replicate resync \
mc replicate resync start \
--remote-bucket "arn:minio:replication::d3c086c7-1d64-40c2-954b-fe8222907033:mydata" \
myminio/mydata
@ -57,9 +57,9 @@ for more information on active-active replication:
.. code-block:: shell
:class: copyable
mc [GLOBALFLAGS] replicate resync \
--remote-bucket "string" \
[--older-than "string"] \
mc [GLOBALFLAGS] replicate resync start|status \
--remote-bucket "string" \
[--older-than "string"] \
ALIAS
.. include:: /includes/common-minio-mc.rst
@ -70,19 +70,40 @@ Parameters
~~~~~~~~~~
.. mc-cmd:: ALIAS
:required:
*Required* the :ref:`alias <alias>` of the MinIO deployment and full path to
The :ref:`alias <alias>` of the MinIO deployment and full path to
the bucket or bucket prefix which MinIO uses as the replication source. For
example:
example, the following command starts replication using the ``data``
bucket on the MinIO deployment associated to the ``primary`` alias.
.. code-block:: none
mc replicate resync myminio/mybucket
mc replicate resync start primary/data --remote-bucket "ARN"
.. mc-cmd:: start
:required:
Starts the resynchronization procedure using the specified
:mc-cmd:`bucket <mc replicate resync ALIAS>` as the source and the
:mc-cmd:`--remote-bucket <mc replicate resync --remote-bucket>` as the
remote target.
Mutually exclusive with :mc-cmd:`mc replicate resync status`.
.. mc-cmd:: status
:required:
Returns the status of resynchronization on the specified
:mc-cmd:`bucket <mc replicate resync ALIAS>` to all remote targets.
Include the :mc-cmd:`~mc replicate resync --remote-bucket` argument to
filter the status output to only the specified remote target.
.. mc-cmd:: --remote-bucket
:required:
*Required* Specify the ARN for the destination deployment and bucket. You
Specify the ARN for the destination deployment and bucket. You
can retrieve the ARN using :mc-cmd:`mc admin bucket remote`:
- Use the :mc-cmd:`mc admin bucket remote ls` to retrieve a list of
@ -92,10 +113,13 @@ Parameters
for the bucket on the destination deployment.
.. mc-cmd:: older-than
:optional:
*Optional* Specify a duration in days where MinIO only resynchronizes
Specify a duration in days where MinIO only resynchronizes
objects older than the specified duration.
Only valid with :mc-cmd:`mc replicate resync start`.
Global Flags
~~~~~~~~~~~~
@ -116,9 +140,9 @@ replication status:
.. code-block:: shell
:class: copyable
mc replicate resync --remote-bucket "arn:minio:replication::UUID:mybucket" myminio/mybucket
mc replicate resync start --remote-bucket "arn:minio:replication::UUID:data" primary/data
- Replace ``myminio/mybucket`` with the :mc-cmd:`~mc replicate add ALIAS` and
- Replace ``primary/data`` with the :mc-cmd:`~mc replicate add ALIAS` and
full bucket path for which to create the replication configuration.
- Replace the :mc-cmd:`~mc replicate add --remote-bucket` value with the

117
source/replication/enable-server-side-multi-site-bucket-replication.rst
View File

@ -8,7 +8,7 @@ Enable Multi-Site Server-Side Bucket Replication
.. contents:: Table of Contents
:local:
:depth: 2
:depth: 1
The procedure on this page configures automatic server-side bucket
@ -61,38 +61,6 @@ MinIO multi-site replication requires MinIO server
Requirements
------------
Replication Requires MinIO Remote Targets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO server-side replication only works between MinIO deployments. All
deployments participating in the multi-site replication configuration
*must* run MinIO. MinIO strongly recommends using the *same* MinIO server
version across all sites.
To configure replication between arbitrary S3-compatible services,
use :mc-cmd:`mc mirror`.
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO relies on the immutability protections provided by versioning to
synchronize objects as part of replication.
Use the :mc-cmd:`mc version suspend` command to enable versioning for the bucket
across *all* MinIO deployments participating in the multi-site replication
configuration.
.. code-block:: shell
:class: copyable
mc version ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc version ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO deployment.
- Replace :mc-cmd:`PATH <mc version ALIAS>` with the bucket on which
to enable versioning.
Install and Configure ``mc`` with Access to Both Clusters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -142,11 +110,6 @@ destination deployments to configure and enable replication rules.
source cluster. You can restrict the user policy to specific buckets
as-needed.
Use the :mc-cmd:`mc admin policy add` to add this policy to *both*
deployments. You can then create a user on both deployments using
:mc-cmd:`mc admin user add` and associate the policy to those users
with :mc-cmd:`mc admin policy set`.
.. tab-item:: Replication Remote User
The following policy provides permissions for enabling synchronization of
@ -170,15 +133,41 @@ destination deployments to configure and enable replication rules.
buckets as an element in the ``Resource`` array similar to
``"arn:aws:s3:::bucketName/*"``.
Use the :mc-cmd:`mc admin policy add` to add this policy to *both*
deployments. You can then create a user on both deployments using
:mc-cmd:`mc admin user add` and associate the policy to those users
with :mc-cmd:`mc admin policy set`.
Use the reference policies above to create the necessary policies using :mc-cmd:`mc admin policy add`. After creating the necessary policy, MinIO strongly recommends creating a dedicated user or service account for supporting bucket replication operations.
MinIO strongly recommends creating users specifically for supporting
bucket replication operations. See
:mc:`mc admin user` and :mc:`mc admin policy` for more complete
documentation on adding users and policies to a MinIO cluster.
- MinIO deployments configured for :ref:`Active Directory/LDAP <minio-external-identity-management-ad-ldap>` or :ref:`OpenID Connect <minio-external-identity-management-openid>` user management can create a dedicated :ref:`service account <minio-idp-service-account>` for bucket replication.
- MinIO deployments configured for :ref:`MinIO <minio-users>` user management can create either a dedicated user *or* service account for bucket replication.
See :mc:`mc admin user`, :mc:`mc admin user svcacct`, and :mc:`mc admin policy` for more complete documentation on adding users, service accounts, and policies to a MinIO cluster.
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
@ -201,10 +190,7 @@ behaviors to ensure consistent and predictable synchronization of objects.
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
:minio-git:`RELEASE.2021-06-07T21-40-51Z
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
MinIO supports automatically
replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the
@ -238,37 +224,6 @@ MinIO does *not* replicate delete operations resulting from the application of
for the bucket on all replication sites to ensure consistent application of
object expiration.
Replication of Encrypted Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects encrypted with automatic
Server-Side Encryption (SSE-S3). Both the source and destination buckets
*must* have automatic SSE-S3 enabled for MinIO to replicate an encrypted object.
As part of the replication process, MinIO *decrypts* the object on the source
bucket and transmits the unencrypted object. The destination MinIO cluster then
re-encrypts the object using the destination bucket SSE-S3 configuration. MinIO
*strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and
destination deployments to ensure the safety of objects during transmission.
MinIO does *not* support replicating client-side encrypted objects
(SSE-C).
Replication of Locked Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects held under
:ref:`WORM Locking <minio-object-locking>`. Both replication buckets *must* have
object locking enabled for MinIO to replicate the locked object. For
active-active configuration, MinIO recommends using the *same*
retention rules on both buckets to ensure consistent behavior across
sites.
You must enable object locking during bucket creation as per S3 behavior.
You can then configure object retention rules at any time.
Object locking requires :ref:`versioning <minio-bucket-versioning>` and
enables the feature implicitly.
Procedure
---------

115
source/replication/enable-server-side-one-way-bucket-replication.rst
View File

@ -8,7 +8,7 @@ Enable One-Way Server-Side Bucket Replication
.. contents:: Table of Contents
:local:
:depth: 2
:depth: 1
The procedure on this page creates a new bucket replication rule for
@ -45,35 +45,6 @@ one-way synchronization of objects between MinIO buckets.
Requirements
------------
MinIO Server-Side Replication Requires a MinIO Cluster as the Destination
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO server-side replication only works between MinIO clusters. Both the
source and destination clusters *must* run MinIO.
To configure replication between arbitrary S3-compatible services,
use :mc-cmd:`mc mirror`.
Enable Versioning on Source and Destination Buckets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO relies on the immutability protections provided by versioning to
synchronize objects between the source and replication target.
Use the :mc-cmd:`mc version suspend` command to enable versioning on
*both* the source and destination bucket before starting this procedure:
.. code-block:: shell
:class: copyable
mc version ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc version ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO cluster.
- Replace :mc-cmd:`PATH <mc version ALIAS>` with the bucket on which
to enable versioning.
Install and Configure ``mc`` with Access to Both Clusters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -123,11 +94,6 @@ source and destination clusters:
source cluster. You can restrict the user policy to specific buckets
as-needed.
Use the :mc-cmd:`mc admin policy add` to add this policy to the
source cluster. Use :mc-cmd:`mc admin user add` to create a user
on the source cluster and :mc-cmd:`mc admin policy set` to associate
the policy to that new user.
.. tab-item:: Replication Remote User
The following policy provides permissions for enabling synchronization of
@ -149,16 +115,42 @@ source and destination clusters:
cluster. To restrict the policy to specific buckets, specify those
buckets as an element in the ``Resource`` array similar to
``"arn:aws:s3:::bucketName/*"``.
Use the :mc-cmd:`mc admin policy add` to add this policy to the
destination cluster. Use :mc-cmd:`mc admin user add` to create a user
on the destination cluster and :mc-cmd:`mc admin policy set` to associate
the policy to that new user.
MinIO strongly recommends creating users specifically for supporting
bucket replication operations. See
:mc:`mc admin user` and :mc:`mc admin policy` for more complete
documentation on adding users and policies to a MinIO cluster.
Use the reference policies above to create the necessary policies using :mc-cmd:`mc admin policy add`. After creating the necessary policy, MinIO strongly recommends creating a dedicated user or service account for supporting bucket replication operations.
- MinIO deployments configured for :ref:`Active Directory/LDAP <minio-external-identity-management-ad-ldap>` or :ref:`OpenID Connect <minio-external-identity-management-openid>` user management can create a dedicated :ref:`service account <minio-idp-service-account>` for bucket replication.
- MinIO deployments configured for :ref:`MinIO <minio-users>` user management can create either a dedicated user *or* service account for bucket replication.
See :mc:`mc admin user`, :mc:`mc admin user svcacct`, and :mc:`mc admin policy` for more complete documentation on adding users, service accounts, and policies to a MinIO cluster.
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
@ -166,11 +158,7 @@ Considerations
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
:minio-git:`RELEASE.2021-06-07T21-40-51Z
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
replicating existing objects in a bucket.
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the
:mc-cmd:`mc replicate add --replicate` or
@ -203,35 +191,6 @@ application of :ref:`lifecycle management expiration rules
See :ref:`minio-replication-behavior-delete` for more complete documentation.
Replication of Encrypted Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects encrypted with automatic
Server-Side Encryption (SSE-S3). Both the source and destination buckets
*must* have automatic SSE-S3 enabled for MinIO to replicate an encrypted object.
As part of the replication process, MinIO *decrypts* the object on the source
bucket and transmits the unencrypted object. The destination MinIO cluster then
re-encrypts the object using the destination bucket SSE-S3 configuration. MinIO
*strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and
destination clusters to ensure the safety of objects during transmission.
MinIO does *not* support replicating client-side encrypted objects
(SSE-C).
Replication of Locked Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects held under
:ref:`WORM Locking <minio-object-locking>`. Both the source and destination
buckets *must* have object locking enabled for MinIO to replicate the locked
object.
You must enable object locking during bucket creation as per S3 behavior.
You can then configure object retention rules at any time.
Object locking requires :ref:`versioning <minio-bucket-versioning>` and
enables the feature implicitly.
Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~

114
source/replication/enable-server-side-two-way-bucket-replication.rst
View File

@ -8,7 +8,7 @@ Enable Two-Way Server-Side Bucket Replication
.. contents:: Table of Contents
:local:
:depth: 2
:depth: 1
The procedure on this page creates a new bucket replication rule for two-way
@ -50,35 +50,6 @@ or more MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`
Requirements
------------
Replication Requires MinIO Remote Targets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO server-side replication only works between MinIO clusters. Both the
source and destination clusters *must* run MinIO.
To configure replication between arbitrary S3-compatible services,
use :mc-cmd:`mc mirror`.
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO relies on the immutability protections provided by versioning to
synchronize objects between the source and replication target.
Use the :mc-cmd:`mc version enable` command to enable versioning on
*both* the source and destination bucket before starting this procedure:
.. code-block:: shell
:class: copyable
mc version ALIAS/PATH
- Replace :mc-cmd:`ALIAS <mc version ALIAS>` with the
:mc:`alias <mc alias>` of the MinIO cluster.
- Replace :mc-cmd:`PATH <mc version ALIAS>` with the bucket on which
to enable versioning.
Install and Configure ``mc`` with Access to Both Clusters.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -128,11 +99,6 @@ destination clusters to configure and enable replication rules.
source cluster. You can restrict the user policy to specific buckets
as-needed.
Use the :mc-cmd:`mc admin policy add` to add this policy to *both*
clusters. You can then create a user on both clusters using
:mc-cmd:`mc admin user add` and associate the policy to those users
with :mc-cmd:`mc admin policy set`.
.. tab-item:: Replication Remote User
The following policy provides permissions for enabling synchronization of
@ -156,15 +122,41 @@ destination clusters to configure and enable replication rules.
buckets as an element in the ``Resource`` array similar to
``"arn:aws:s3:::bucketName/*"``.
Use the :mc-cmd:`mc admin policy add` to add this policy to *both*
clusters. You can then create a user on both clusters using
:mc-cmd:`mc admin user add` and associate the policy to those users
with :mc-cmd:`mc admin policy set`.
Use the reference policies above to create the necessary policies using :mc-cmd:`mc admin policy add`. After creating the necessary policy, MinIO strongly recommends creating a dedicated user or service account for supporting bucket replication operations.
MinIO strongly recommends creating users specifically for supporting
bucket replication operations. See
:mc:`mc admin user` and :mc:`mc admin policy` for more complete
documentation on adding users and policies to a MinIO cluster.
- MinIO deployments configured for :ref:`Active Directory/LDAP <minio-external-identity-management-ad-ldap>` or :ref:`OpenID Connect <minio-external-identity-management-openid>` user management can create a dedicated :ref:`service account <minio-idp-service-account>` for bucket replication.
- MinIO deployments configured for :ref:`MinIO <minio-users>` user management can create either a dedicated user *or* service account for bucket replication.
See :mc:`mc admin user`, :mc:`mc admin user svcacct`, and :mc:`mc admin policy` for more complete documentation on adding users, service accounts, and policies to a MinIO cluster.
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
@ -188,10 +180,7 @@ objects.
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting with :mc:`mc` :minio-git:`RELEASE.2021-06-13T17-48-22Z
<mc/releases/tag/RELEASE.2021-06-13T17-48-22Z>` and :mc:`minio`
:minio-git:`RELEASE.2021-06-07T21-40-51Z
<minio/releases/tag/RELEASE.2021-06-07T21-40-51Z>`, MinIO supports automatically
MinIO supports automatically
replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the
@ -227,37 +216,6 @@ of object expiration.
See :ref:`minio-replication-behavior-delete` for more complete documentation.
Replication of Encrypted Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects encrypted with automatic
Server-Side Encryption (SSE-S3). Both the source and destination buckets
*must* have automatic SSE-S3 enabled for MinIO to replicate an encrypted object.
As part of the replication process, MinIO *decrypts* the object on the source
bucket and transmits the unencrypted object. The destination MinIO cluster then
re-encrypts the object using the destination bucket SSE-S3 configuration. MinIO
*strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and
destination clusters to ensure the safety of objects during transmission.
MinIO does *not* support replicating client-side encrypted objects
(SSE-C).
Replication of Locked Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating objects held under
:ref:`WORM Locking <minio-object-locking>`. Both replication buckets *must* have
object locking enabled for MinIO to replicate the locked object. For
active-active configuration, MinIO recommends using the *same*
retention rules on both buckets to ensure consistent behavior across
sites.
You must enable object locking during bucket creation as per S3 behavior.
You can then configure object retention rules at any time.
Object locking requires :ref:`versioning <minio-bucket-versioning>` and
enables the feature implicitly.
Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~

2
source/replication/replication-overview.rst
View File

@ -312,7 +312,7 @@ The replication process generally has one of the following flows:
/replication/enable-server-side-one-way-bucket-replication
/replication/enable-server-side-two-way-bucket-replication
/replication/enable-server-side-multi-site-bucket-replication
/replication/server-side-replication-resynchronize-remote
.. _minio-bucket-replication-clientside:

200
source/replication/server-side-replication-resynchronize-remote.rst Normal file
View File

@ -0,0 +1,200 @@
.. _minio-bucket-replication-resynchronize:
========================================
Resynchronize Bucket from Remote Replica
========================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
The procedure on this page resynchronizes the contents of a MinIO
bucket using a healthy replication remote. Resynchronization supports
recovery after partial or total loss of data on a MinIO deployment in a
replica configuration.
For example, consider a MinIO active-active replication configuration similar
to the following:
.. image:: /images/replication/active-active-twoway-replication.svg
:width: 600px
:alt: Active-Active Replication synchronizes data between two remote deployments.
:align: center
Resynchronization allows using the healthy data on one of the participating
MinIO deployments as the source for rebuilding the other deployment.
Resynchronization is a per-bucket process. You must repeat resynchronization
for each bucket on the remote which suffered partial or total data loss.
.. admonition:: Professional Support during BC/DR Operations
:class: important
`MinIO SUBNET <https://min.io/pricing?jmp=docs>`__ users can `log in <https://subnet.min.io/>`__ and create a new issue related to resynchronization. Coordination with MinIO Engineering via SUBNET can ensure successful resynchronization and restoration of normal operations, including performance testing and health diagnostics.
Community users can seek support on the `MinIO Community Slack <https://minio.slack.com>`__. Community Support is best-effort only and has no SLAs around responsiveness.
.. _minio-bucket-replication-serverside-resynchronize-requirements:
Requirements
------------
MinIO Deployments Must Be Online
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization requires both the source and remote deployments be online and
able to accept read and write operations. The source *must* have
complete network connectivity to the remote.
The remote deployment may be "unhealthy" in that it has suffered partial or
total data loss. Resynchronization addresses the data loss as long as both
source and destination maintain connectivity.
Resynchronization Requires Existing Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization requires the healthy source deployment have an existing replication configuration for the unhealthy target bucket. Additionally, resynchronization only applies to those replication rules created with the :ref:`existing object replication <minio-replication-behavior-existing-objects>` option.
- Use :mc-cmd:`mc admin bucket remote ls` to review the configured remote targets on the healthy source bucket.
- Use :mc-cmd:`mc replicate ls` to review the configured replication rules on the healthy source bucket.
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Resynchronization Requires Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Resynchronization is a background processes that continually checks objects in
the source MinIO bucket and copies them to the remote as-needed. The time
required for replication to complete may vary depending on the number and size
of objects, the throughput to the remote MinIO deployment, and the load on the
source MinIO deployment. Total time for completion is generally not predictable
due to these variables.
MinIO recommends configuring load balancers or proxies to direct traffic only
to the healthy cluster until synchronization completes. The following commands
can provide insight into the resynchronization status:
- :mc-cmd:`mc replicate status` on the source and remote to track total
replicated data.
- Run ``mc ls -r --versions ALIAS/BUCKET | wc -l`` against both source and
remote to validate the total number of objects and object versions on each.
Resynchronize Objects after Data Loss
-------------------------------------
This procedure uses an existing :ref:`MinIO replication configuration <minio-bucket-replication-serverside>` to restore missing data to one of the MinIO deployments participating in that configuration. Specifically, a healthy MinIO deployment (the ``SOURCE``) synchronizes it's existing data to the unhealthy MinIO deployment (the ``TARGET``).
This procedure assumes an existing :ref:`alias <alias>` for the ``SOURCE`` that has the :ref:`necessary permissions <minio-bucket-replication-serverside-twoway-permissions>` for configuring replication.
You can repeat this procedure for each bucket that requires resynchronization. You can have no more than one replication job running per bucket.
1) List the Configured Replication Targets on the Healthy Source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run the :mc-cmd:`mc admin bucket remote ls` command to list the configured remote targets on the healthy ``SOURCE`` deployment for the ``BUCKET`` that requires resynchronization.
.. code-block:: shell
:class: copyable
mc admin bucket remote ls SOURCE/BUCKET
- Replace ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
- Replace ``BUCKET`` with the name of the bucket to use as the source for resynchronization.
The output resembles the following:
.. code-block:: shell
Remote URL Source->Target ARN SYNC PROXY
https://minio-2.example.net:9000 data ->data arn:minio:replication::UUID:BUCKET proxy
Identify the ARN associated to the ``BUCKET`` on the unhealthy ``TARGET`` MinIO deployment for use in the next step.
2) Start the Resynchronization Procedure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run the :mc-cmd:`mc replicate resync start` command to begin the resynchronization process:
.. code-block:: shell
:class: copyable
mc replicate resync start --remote-bucket "arn:minio:replication::UUID:BUCKET" SOURCE/BUCKET
- Replace the ``--remote-bucket`` value with the ARN of the unhealthy ``BUCKET`` on the ``TARGET`` MinIO deployment.
- Replaced ``SOURCE`` with the :ref:`alias <alias>` of the source MinIO deployment.
- Replace the ``BUCKET`` with the name of the bucket on the healthy ``SOURCE`` MinIO
deployment.
The command returns a resynchronization job ID indicating that the process has
begun.
3) Monitor Resynchronization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate resync status` command on the source deployment to track the received replication data:
.. code-block:: shell
:class: copyable
mc replicate resync status ALIAS/BUCKET
The output resembles the following:
.. code-block:: shell
mc replicate resync status /data
Resync status summary:
● arn:minio:replication::6593d572-4dc3-4bb9-8d90-7f79cc612f01:data
Status: Ongoing
Replication Status | Size (Bytes) | Count
Replicated | 2.3 GiB | 18
Failed | 0 B | 0
The :guilabel:`Status` updates to ``Completed`` once the resynchronization
process completes.
4) Next Steps
~~~~~~~~~~~~~
- If the ``TARGET`` bucket damage extends to replication rules, you must recreate those rules to match the previous replication configuration. See :ref:`minio-bucket-replication-serverside-twoway` for additional guidance.
- Perform basic validation that all buckets in the replication configuration show similar results for commands such as :mc-cmd:`mc ls` and :mc-cmd:`mc stat`.
- After restoring any replication rules and verifying replication between sites, you can configure the reverse proxy, load balancer, or other network control plane managing connections to resume sending traffic to the resynchronized deployment.