1
0
mirror of https://github.com/minio/docs.git synced 2025-07-28 19:42:10 +03:00

Simplify Bucket Replication Docs (#513)

* Updates and simplifies bucket replication docs
* Improves doc structure
* Updates to corrected images
* Moving common bucket replication content to includes file
* Adds MinIO Console instruction
This commit is contained in:
Daryl White
2022-07-29 17:52:13 -05:00
committed by GitHub
parent c783e10594
commit 8c727d7f90
12 changed files with 2598 additions and 2939 deletions

View File

@ -314,6 +314,7 @@ The replication process generally has one of the following flows:
:hidden: :hidden:
:titlesonly: :titlesonly:
/administration/bucket-replication/bucket-replication-requirements
/administration/bucket-replication/enable-server-side-one-way-bucket-replication /administration/bucket-replication/enable-server-side-one-way-bucket-replication
/administration/bucket-replication/enable-server-side-two-way-bucket-replication /administration/bucket-replication/enable-server-side-two-way-bucket-replication
/administration/bucket-replication/enable-server-side-multi-site-bucket-replication /administration/bucket-replication/enable-server-side-multi-site-bucket-replication

View File

@ -0,0 +1,62 @@
.. _minio-bucket-replication-requirements:
=========================================
Requirements to Set Up Bucket Replication
=========================================
.. default-domain:: minio
.. contents:: Table of Contents
:local:
:depth: 1
.. _minio-bucket-replication-serverside-oneway-permissions:
Bucket replication uses rules to synchronize the contents of a bucket on one MinIO deployment to a bucket on a remote MinIO deployment.
Replication can be done in any of the following ways:
- :ref:`Active-Passive <minio-bucket-replication-serverside-oneway>`
Eligible objects replicate from the source bucket to the remote bucket.
Any changes on the remote bucket do not replicate back.
- :ref:`Active-Active <minio-bucket-replication-serverside-twoway>`
Changes to eligible objects of either bucket replicate to the other bucket in a two-way direction.
- :ref:`Multi-Site Active-Active <minio-bucket-replication-serverside-multi>`
Changes to eligible objects on any bucket set up for bucket replication replicte to all of the other buckets.
Ensure you meet the following prerequisites before you set up any of these replication configurations.
Permissions Required for Setting Up Bucket Replication
------------------------------------------------------
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Matching Object Encryption Settings for Bucket Replication
----------------------------------------------------------
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Bucket Replication Requires MinIO Deployments
---------------------------------------------
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Versioning Objects for Bucket Replication
-----------------------------------------
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Matching Object Locking State With Bucket Replication
-----------------------------------------------------
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking

View File

@ -8,7 +8,7 @@ Enable Multi-Site Server-Side Bucket Replication
.. contents:: Table of Contents .. contents:: Table of Contents
:local: :local:
:depth: 1 :depth: 2
The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the :ref:`minio-bucket-replication-serverside-twoway` procedure with additional considerations required to ensure predictable replication behavior across all sites. The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the :ref:`minio-bucket-replication-serverside-twoway` procedure with additional considerations required to ensure predictable replication behavior across all sites.
@ -39,12 +39,22 @@ Multi-Site Active-Active replication configurations can span multiple racks, dat
Requirements Requirements
------------ ------------
Install and Configure ``mc`` with Access to Both Clusters. You must meet all of the basic requirements for bucket replication described in :ref:`Bucket Replication Requirements <minio-bucket-replication-requirements>`.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination deployments. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``. In addition, to create multi-site bucket replication set up, you must meet the following additional requirements:
Use the :mc:`mc alias` command to create an alias for both MinIO deployments. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*: Access to All Clusters
~~~~~~~~~~~~~~~~~~~~~~
You must have network access and log in credentials with correct permissions to all deployments to set up multi-site active-active bucket replication.
You can access the deployments by logging in to the :ref:`MinIO Console <minio-console>` for each deployment or by installing :mc:`mc` and using the command line.
If using the command line, use the :mc:`mc alias` command to create an alias for each MinIO deployment.
Alias creation requires specifying an access key for a user on the deployment.
This user **must** have permission to create and manage users and policies on the deployment.
Specifically, ensure the user has *at minimum*:
- :policy-action:`admin:CreateUser` - :policy-action:`admin:CreateUser`
- :policy-action:`admin:ListUsers` - :policy-action:`admin:ListUsers`
@ -53,178 +63,151 @@ Use the :mc:`mc alias` command to create an alias for both MinIO deployments. Al
- :policy-action:`admin:GetPolicy` - :policy-action:`admin:GetPolicy`
- :policy-action:`admin:AttachUserOrGroupPolicy` - :policy-action:`admin:AttachUserOrGroupPolicy`
.. _minio-bucket-replication-serverside-multi-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations Considerations
-------------- --------------
Use Consistent Replication Settings Click to expand any of the following:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports customizing the replication configuration to enable or disable the following replication behaviors: .. dropdown:: Use Consistent Replication Settings
:icon: fold-down
- Replication of delete operations MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:
- Replication of delete markers
- Replication of existing objects
- Replication of metadata-only changes
When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects. - Replication of delete operations
- Replication of delete markers
- Replication of existing objects
- Replication of metadata-only changes
Replication of Existing Objects When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports automatically replicating existing objects in a bucket. .. dropdown:: Replication of Existing Objects
:icon: fold-down
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects. MinIO supports automatically replicating existing objects in a bucket.
Replication of Delete Operations MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This procedure includes the required flags for enabling replication of existing objects.
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects: .. dropdown:: Replication of Delete Operations
:icon: fold-down
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket. MinIO supports replicating delete operations onto the target bucket.
Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket. - For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers. - For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration. MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`.
This procedure includes the required flags for enabling replication of delete operations and delete markers.
Procedure MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration.
Procedure
--------- ---------
This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure *before* attempting to implement the documented steps. This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure *before* attempting to implement the documented steps.
- :ref:`Configure Multi-Site Bucket Replication Using the MinIO Console <minio-bucket-replication-multi-site-minio-console-procedure>`
- :ref:`Create the Replication Rules <minio-bucket-replication-multi-site-minio-console-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-multi-site-minio-console-validate-replication-config>`
- :ref:`Configure Multi-Site Bucket Replication Using the Command Line <minio-bucket-replication-multi-site-minio-cli-procedure>`
- :ref:`Create Replication Remote Targets <minio-bucket-replication-multi-site-minio-cli-create-remote-targets>`
- :ref:`Create New Bucket Replication Rules <minio-bucket-replication-multi-site-minio-cli-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-multi-site-minio-cli-verify-replication-config>`
.. _minio-bucket-replication-multi-site-minio-console-procedure:
Configure Multi-Site Bucket Replication Using the MinIO Console
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _minio-bucket-replication-multi-site-minio-console-create-replication-rules:
1) Create the Replication Rules
+++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-create-bucket-replication-rule-console-desc
:end-before: end-create-bucket-replication-rule-console-desc
Repeat the above steps to create a rule from this deployment to each of the other target deployments.
Then, repeat the above steps on each of the other deployments in the multi-site setup so that each deployment has a separate replication rule for all of the other deployments.
.. _minio-bucket-replication-multi-site-minio-console-validate-replication-config:
2) Validate the Replication Configuration
+++++++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-console-desc
:end-before: end-validate-bucket-replication-console-desc
Repeat this test on each deployment by copying a new unique file and checking that the file replicates to each of the other deployments.
.. _minio-bucket-replication-multi-site-minio-cli-procedure:
Configure Multi-Site Bucket Replication Using the Command Line (:mc:`mc`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses the placeholder ``ALIAS`` to reference the :ref:`alias <alias>` each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment. This procedure uses the placeholder ``ALIAS`` to reference the :ref:`alias <alias>` each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-multi-permissions>`. This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-multi-permissions>`.
1) Create the Replication Remote Target .. _minio-bucket-replication-multi-site-minio-cli-create-remote-targets:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket. 1) Create Replication Remote Targets
++++++++++++++++++++++++++++++++++++
.. code-block:: shell .. include:: /includes/common/bucket-replication.rst
:class: copyable :start-after: start-create-replication-remote-targets-cli-desc
:end-before: end-create-replication-remote-targets-cli-desc
mc admin bucket remote add ALIAS/BUCKET \ Repeat these instructions for each remote MinIO deployment participating in the multi-site replication configuration.
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
--service "replication"
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. For example, a multi-site replication configuration consisting of three MinIO deployments ``minio1``, ``minio2``, and ``minio3`` requires repeating this step twice on each deployment. Specifically:
- Replace ``HOSTNAME`` with the URL of the remote MinIO deployment. - The ``minio1`` deployment requires defining separate remote targets for ``minio2`` and for ``minio3``.
- Replace ``BUCKET`` with the name of the bucket on the remote deployment to use as the replication destination. - The ``minio2`` deployment requires defining separate remote targets for ``minio1`` and for ``minio3``.
The command returns an ARN similar to the following: - The ``minio3`` deployment requires defining separate remote targets for ``minio1`` and for ``minio2``.
.. code-block:: shell More than three deployments requires additional remote targets on each deployment to create the required targets for each origin and destination bucket compination.
Role ARN = 'arn:minio:replication::<UUID>:BUCKET' Record the ARN generated for each remote and note which origin-destination bucket combination you generated the ARN for.
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created. .. _minio-bucket-replication-multi-site-minio-cli-create-replication-rules:
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically: 2) Create New Bucket Replication Rules
++++++++++++++++++++++++++++++++++++++
- The ``Alpha`` deployment would perform this step once for .. include:: /includes/common/bucket-replication.rst
``Baker`` and once for ``Charlie``. :start-after: start-create-bucket-replication-rule-cli-desc
:end-before: end-create-bucket-replication-rule-cli-desc
- The ``Baker`` deployment would perform this step once for ``Alpha`` and Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration.
once for ``Charlie``. For example, a multi-site replication configuration consisting of MinIO deployments ``minio1``, ``minio2``, and ``minio3`` would require repeating this step on each deployment for each remote.
Specifically, in this scenario, perform this step twice on each deployment:
- The ``Charlie`` deployment would perform this step once for ``Baker`` and - On the ``minio1`` deployment, once for a rule for ``minio2`` and again for a separate rule for ``minio3``.
once for ``Alpha``.
2) Create a New Bucket Replication Rule - On the ``minio2`` deployment, once for a rule for ``minio1`` and again for a separate rule for ``minio3``.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate add` command to add the new server-side - On the ``minio3`` deployment, once for a rule for ``minio1`` and again for a separate rule for ``minio2``.
replication rule to the each MinIO deployment.
.. code-block:: shell .. _minio-bucket-replication-multi-site-minio-cli-verify-replication-config:
:class: copyable
mc replicate add ALIAS/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``ALIAS`` deployment to use as the replication source. Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALIAS`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments ``Alpha``, ``Baker``, and ``Charlie`` would require repeating this step on each deployment for each remote. Specifically:
- The ``Alpha`` deployment would perform this step once for
``Baker`` and once for ``Charlie``.
- The ``Baker`` deployment would perform this step once for ``Alpha`` and
once for ``Charlie``.
- The ``Charlie`` deployment would perform this step once for ``Baker`` and
once for ``Alpha``.
3) Validate the Replication Configuration 3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++++++++++
Use :mc-cmd:`mc cp` to copy a new object the bucket on any of the deployments: .. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-cli-desc
:end-before: end-validate-bucket-replication-cli-desc
.. code-block:: shell Repeat this test on each deployment by copying a new unique file and checking that the file replicates to each of the other deployments.
:class: copyable
mc cp ~/foo.txt ALIAS/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on each remote deployment:
.. code-block:: shell
:class: copyable
mc ls REMOTE/BUCKET
Repeat this test on each of the deployments by copying a new unique file and checking the other deployments for that file.
You can also use :mc-cmd:`mc stat` to check the file to check the current :ref:`replication stage <minio-replication-process>` of the object. You can also use :mc-cmd:`mc stat` to check the file to check the current :ref:`replication stage <minio-replication-process>` of the object.

View File

@ -8,21 +8,145 @@ Enable One-Way Server-Side Bucket Replication
.. contents:: Table of Contents .. contents:: Table of Contents
:local: :local:
:depth: 1 :depth: 2
The procedure on this page creates a new bucket replication rule for one-way synchronization of objects between MinIO buckets. The procedure on this page creates a new bucket replication rule for one-way synchronization of objects from one MinIO bucket to another MinIO bucket.
The buckets can be on the same MinIO deployment or on separate MinIO deployments.
.. image:: /images/replication/active-passive-oneway-replication.svg .. image:: /images/replication/active-passive-oneway-replication.svg
:width: 450px :width: 800px
:alt: Active-Passive Replication synchronizes data from a source MinIO cluster to a remote MinIO cluster. :alt: Active-Passive Replication synchronizes data from a source MinIO deployment to a remote MinIO deployment.
:align: center :align: center
- To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
- To configure two-way "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-twoway`. - To configure two-way "active-active" replication between MinIO buckets, see :ref:`minio-bucket-replication-serverside-twoway`.
- To configure multi-site "active-active" replication between MinIO deployments, see :ref:`minio-bucket-replication-serverside-multi`
- To configure multi-site "active-active" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi` .. note::
To configure replication between arbitrary S3-compatible services (not necessarily MinIO), use :mc-cmd:`mc mirror`.
Requirements
------------
Replication requires all participating clusters meet the :ref:`following requirements <minio-bucket-replication-requirements>`.
This procedure assumes you have reviewed and validated those requirements.
For more details, see the :ref:`Bucket Replication Requirements <minio-bucket-replication-requirements>` page.
Considerations
--------------
Click to expand any of the following:
.. dropdown:: Replication of Existing Objects
:icon: fold-down
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag.
This procedure includes the required flags for enabling replication of existing objects.
.. dropdown:: Replication of Delete Operations
:icon: fold-down
MinIO supports replicating S3 ``DELETE`` operations onto the target bucket.
Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`.
This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
See :ref:`minio-replication-behavior-delete` for more complete documentation.
.. dropdown:: Multi-Site Replication
:icon: fold-down
MinIO supports configuring multiple remote targets per bucket or bucket prefix.
For example, you can configure a bucket to replicate data to two or more remote MinIO deployments, where one deployment is a 1:1 copy (replication of all operations including deletions) and another is a full historical record (replication of only non-destructive write operations).
This procedure documents one-way replication to a single remote MinIO deployment.
You can repeat this tutorial to replicate a single bucket to multiple remote targets.
Procedure
---------
- :ref:`Configure One-Way Bucket Replication Using the MinIO Console <minio-bucket-replication-one-way-minio-console-procedure>`
- :ref:`Create a New Bucket Replication Rule <minio-bucket-replication-one-way-minio-console-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-one-way-minio-console-validate-replication-config>`
- :ref:`Configure One-Way Bucket Replication Using the Command Line <minio-bucket-replication-one-way-minio-cli-procedure>`
- :ref:`Create a Replication Remote Target <minio-bucket-replication-one-way-minio-cli-create-remote-targets>`
- :ref:`Create a New Bucket Replication Rule <minio-bucket-replication-one-way-minio-cli-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-one-way-minio-cli-verify-replication-config>`
.. _minio-bucket-replication-one-way-minio-console-procedure:
Configure One-Way Bucket Replication Using the MinIO Console
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _minio-bucket-replication-one-way-minio-console-create-replication-rules:
1) Create a New Bucket Replication Rule
+++++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-create-bucket-replication-rule-console-desc
:end-before: end-create-bucket-replication-rule-console-desc
.. _minio-bucket-replication-one-way-minio-console-validate-replication-config:
2) Validate the Replication Configuration
+++++++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-console-desc
:end-before: end-validate-bucket-replication-console-desc
.. _minio-bucket-replication-one-way-minio-cli-procedure:
Configure One-Way Bucket Replication Using the Command Line (:mc:`mc`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses the :ref:`aliases <alias>` ``SOURCE`` and ``REMOTE`` to reference each MinIO deployment being configured for replication.
Replace these values with the appropriate alias for your target MinIO deployments.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-oneway-permissions>`.
.. _minio-bucket-replication-one-way-minio-cli-create-remote-targets:
1) Create a Replication Remote Target
+++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-create-replication-remote-targets-cli-desc
:end-before: end-create-replication-remote-targets-cli-desc
.. _minio-bucket-replication-one-way-minio-cli-create-replication-rules:
2) Create a New Bucket Replication Rule
+++++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-create-bucket-replication-rule-cli-desc
:end-before: end-create-bucket-replication-rule-cli-desc
.. _minio-bucket-replication-one-way-minio-cli-verify-replication-config:
3) Validate the Replication Configuration
+++++++++++++++++++++++++++++++++++++++++
.. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-cli-desc
:end-before: end-validate-bucket-replication-cli-desc
.. seealso:: .. seealso::
@ -31,156 +155,3 @@ The procedure on this page creates a new bucket replication rule for one-way syn
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule. - Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule. - Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. _minio-bucket-replication-serverside-oneway-requirements:
Requirements
------------
.. _minio-bucket-replication-serverside-oneway-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations
--------------
Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
MinIO supports replicating S3 ``DELETE`` operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
See :ref:`minio-replication-behavior-delete` for more complete documentation.
Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~
MinIO supports configuring multiple remote targets per bucket or bucket prefix. For example, you can configure a bucket to replicate data to two or more remote MinIO deployments, where one deployment is a 1:1 copy (replication of all operations including deletions) and another is a full historical record (replication of only non-destructive write operations).
This procedure documents one-way replication to a single remote MinIO deployment. You can repeat this tutorial for multiple remote targets for a single bucket.
Procedure
---------
This procedure uses the :ref:`aliases <alias>` ``SOURCE`` and ``REMOTE`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments.
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-oneway-permissions>`.
1) Create the Replication Remote Target
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the destination cluster. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.
.. code-block:: shell
:class: copyable
mc admin bucket remote add SOURCE/BUCKET \
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
--service "replication"
[--sync]
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication.
- Replace ``HOSTNAME`` with the URL of the ``REMOTE`` cluster.
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination.
- Include the :mc-cmd:`~mc admin bucket remote add --sync` option to enable synchronous replication. Omit the option to use the default of asynchronous replication. See the reference documentation for :mc-cmd:`mc admin bucket remote add` for more information on synchronous vs asynchronous replication before using this parameter.
The command returns an ARN similar to the following:
.. code-block:: shell
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
Copy the ARN string for use in the next step.
2) Create a New Bucket Replication Rule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc replicate add` command to add the new server-side
replication rule to the source MinIO cluster.
.. code-block:: shell
:class: copyable
mc replicate add SOURCE/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``SOURCE`` deployment to use as the replication source. Replace ``SOURCE`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step.
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``SOURCE`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
.. code-block:: shell
:class: copyable
mc cp ~/foo.txt SOURCE/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls TARGET/BUCKET

View File

@ -8,13 +8,12 @@ Enable Two-Way Server-Side Bucket Replication
.. contents:: Table of Contents .. contents:: Table of Contents
:local: :local:
:depth: 1 :depth: 2
The procedure on this page creates a new bucket replication rule for two-way "active-active" synchronization of objects between MinIO buckets. The procedure on this page creates a new bucket replication rule for two-way "active-active" synchronization of objects between MinIO buckets.
.. image:: /images/replication/active-active-twoway-replication.svg .. image:: /images/replication/active-active-twoway-replication.svg
:width: 600px :width: 800px
:alt: Active-Active Replication synchronizes data between two remote clusters. :alt: Active-Active Replication synchronizes data between two remote clusters.
:align: center :align: center
@ -26,26 +25,30 @@ The procedure on this page creates a new bucket replication rule for two-way "ac
This tutorial covers configuring Active-Active replication between two MinIO clusters. For a tutorial on multi-site replication between three or more MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`. This tutorial covers configuring Active-Active replication between two MinIO clusters. For a tutorial on multi-site replication between three or more MinIO clusters, see :ref:`minio-bucket-replication-serverside-multi`.
.. seealso::
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
replication rule.
- Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. _minio-bucket-replication-serverside-twoway-requirements: .. _minio-bucket-replication-serverside-twoway-requirements:
Requirements Requirements
------------ ------------
Install and Configure ``mc`` with Access to Both Clusters. You must meet all of the basic requirements for bucket replication described in :ref:`Bucket Replication Requirements <minio-bucket-replication-requirements>`.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This procedure uses :mc:`mc` for performing operations on both the source and destination MinIO cluster. Install :mc:`mc` on a machine with network access to both source and destination clusters. See the ``mc`` :ref:`Installation Quickstart <mc-install>` for instructions on downloading and installing ``mc``. In addition, to set up active-active bucket replication, you must meet the following additional requirements:
Use the :mc:`mc alias` command to create an alias for both MinIO clusters. Alias creation requires specifying an access key for a user on the cluster. This user **must** have permission to create and manage users and policies on the cluster. Specifically, ensure the user has *at minimum*: .. _minio-bucket-replication-serverside-twoway-permissions:
Access to Both Clusters
~~~~~~~~~~~~~~~~~~~~~~~
You must have network access and login credentials with required permissions to both deployment to set up active-active bucket replication.
You can access the deployments by logging in to the :ref:`MinIO Console <minio-console>` for each deployment or by installing :mc:`mc` and using the command line.
If using the command line, use the :mc:`mc alias` command to create an alias for both MinIO deployments.
Alias creation requires specifying an access key for a user on the deployment.
This user **must** have permission to create and manage users and policies on the deployment.
Specifically, ensure the user has *at minimum*:
- :policy-action:`admin:CreateUser` - :policy-action:`admin:CreateUser`
- :policy-action:`admin:ListUsers` - :policy-action:`admin:ListUsers`
@ -54,182 +57,154 @@ Use the :mc:`mc alias` command to create an alias for both MinIO clusters. Alias
- :policy-action:`admin:GetPolicy` - :policy-action:`admin:GetPolicy`
- :policy-action:`admin:AttachUserOrGroupPolicy` - :policy-action:`admin:AttachUserOrGroupPolicy`
.. _minio-bucket-replication-serverside-twoway-permissions:
Required Permissions
~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-required-permissions
:end-before: end-replication-required-permissions
Replication Requires Matching Object Encryption Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-encrypted-objects
:end-before: end-replication-encrypted-objects
Replication Requires MinIO Deployments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-minio-only
:end-before: end-replication-minio-only
Replication Requires Versioning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-versioning
:end-before: end-replication-requires-versioning
Replication Requires Matching Object Locking State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. include:: /includes/common-replication.rst
:start-after: start-replication-requires-object-locking
:end-before: end-replication-requires-object-locking
Considerations Considerations
-------------- --------------
Use Consistent Replication Settings .. dropdown:: Use Consistent Replication Settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :icon: fold-down
MinIO supports customizing the replication configuration to enable or disable MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:
the following replication behaviors:
- Replication of delete operations - Replication of delete operations
- Replication of delete markers - Replication of delete markers
- Replication of existing objects - Replication of existing objects
- Replication of metadata-only changes - Replication of metadata-only changes
When configuring replication rules for a bucket, ensure that both MinIO deployments participating in active-active replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects. When configuring replication rules for a bucket, ensure that both MinIO deployments participating in active-active replication use the *same* replication behaviors to ensure consistent and predictable synchronization of objects.
Replication of Existing Objects .. dropdown:: Replication of Existing Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :icon: fold-down
MinIO supports automatically replicating existing objects in a bucket. MinIO supports automatically replicating existing objects in a bucket.
MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag. This procedure includes the required flags for enabling replication of existing objects. MinIO requires explicitly enabling replication of existing objects using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate` and including the ``existing-objects`` replication feature flag.
This procedure includes the required flags for enabling replication of existing objects.
Replication of Delete Operations .. dropdown:: Replication of Delete Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :icon: fold-down
MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects: MinIO supports replicating delete operations onto the target bucket.
Specifically, MinIO can replicate versioning :s3-docs:`Delete Markers <versioning-workflows.html>` and the deletion of specific versioned objects:
- For delete operations on an object, MinIO replication also creates the delete marker on the target bucket. - For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.
- For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket. - For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.
MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`. This procedure includes the required flags for enabling replication of delete operations and delete markers. MinIO requires explicitly enabling replication of delete operations using the :mc-cmd:`mc replicate add --replicate` or :mc-cmd:`mc replicate edit --replicate`.
This procedure includes the required flags for enabling replication of delete operations and delete markers.
MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`. Configure matching expiration rules on both the source and destination bucket to ensure consistent application of object expiration. MinIO does *not* replicate delete operations resulting from the application of :ref:`lifecycle management expiration rules <minio-lifecycle-management-expiration>`.
Configure matching expiration rules on both the source and destination bucket to ensure consistent application of object expiration.
See :ref:`minio-replication-behavior-delete` for more complete documentation. See :ref:`minio-replication-behavior-delete` for more complete documentation.
Multi-Site Replication .. dropdown:: Multi-Site Replication
~~~~~~~~~~~~~~~~~~~~~~ :icon: fold-down
MinIO supports configuring multiple remote targets per bucket or bucket prefix. This enables configuring multi-site active-active replication between MinIO deployments. MinIO supports configuring multiple remote targets per bucket or bucket prefix.
This enables configuring multi-site active-active replication between MinIO deployments.
This procedure covers active-active replication between *two* MinIO sites. You can repeat this procedure for each "pair" of MinIO deployments in the replication mesh. For a dedicated tutorial, see :ref:`minio-bucket-replication-serverside-multi`. This procedure covers active-active replication between *two* MinIO sites.
You can repeat this procedure for each "pair" of MinIO deployments in the replication mesh. For a dedicated tutorial, see :ref:`minio-bucket-replication-serverside-multi`.
Procedure Procedure
--------- ---------
This procedure uses the :ref:`aliases <alias>` ``ALPHA`` and ``BAKER`` to reference each MinIO deployment being configured for replication. Replace these values with the appropriate alias for your target MinIO deployments. - :ref:`Configure Two-Way Bucket Replication Using the MinIO Console <minio-bucket-replication-two-way-minio-console-procedure>`
- :ref:`Create a New Bucket Replication Rule on Each Deployment <minio-bucket-replication-two-way-minio-console-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-two-way-minio-console-validate-replication-config>`
- :ref:`Configure Two-Way Bucket Replication Using the Command Line <minio-bucket-replication-two-way-minio-cli-procedure>`
- :ref:`Create Replication Remote Targets <minio-bucket-replication-two-way-minio-cli-create-remote-targets>`
- :ref:`Create a New Bucket Replication Rule on Each Deployment <minio-bucket-replication-two-way-minio-cli-create-replication-rules>`
- :ref:`Validate the Replication Configuration <minio-bucket-replication-two-way-minio-cli-verify-replication-config>`
This procedure assumes each alias corresponds to a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-twoway-permissions>`. .. _minio-bucket-replication-two-way-minio-console-procedure:
1) Create the Replication Remote Target Configure Two-Way Bucket Replication Using the MinIO Console
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target for the each deployment. MinIO supports *one* remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket. .. _minio-bucket-replication-two-way-minio-console-create-replication-rules:
.. code-block:: shell 1) Create a New Bucket Replication Rule on Each Deployment
:class: copyable ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mc admin bucket remote add ALPHA/BUCKET \ .. include:: /includes/common/bucket-replication.rst
https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \ :start-after: start-create-bucket-replication-rule-console-desc
--service "replication" :end-before: end-create-bucket-replication-rule-console-desc
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. Repeat the above steps to create a rule in the other direction.
A) Go to the Console for the destination deployment used above.
B) Create a replication rule from the second deployment back to the first deployment.
The first deployment becomes the target deployment for the rule on the second deployment.
- Replace ``HOSTNAME`` with the URL of the ``BAKER`` deployment. .. _minio-bucket-replication-two-way-minio-console-validate-replication-config:
- Replace ``BUCKET`` with the name of the bucket on the ``REMOTE`` deployment to use as the replication destination. 2) Validate the Replication Configuration
+++++++++++++++++++++++++++++++++++++++++
The command returns an ARN similar to the following: .. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-console-desc
:end-before: end-validate-bucket-replication-console-desc
.. code-block:: shell .. _minio-bucket-replication-two-way-minio-cli-procedure:
Role ARN = 'arn:minio:replication::<UUID>:BUCKET' Configure Two-Way Bucket Replication Using the Command Line (:mc:`mc`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created. This procecure creates two-way, active-active replication between two MinIO deployments.
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment. This procedure assumes you have already defined an alias for each deployment as a user with the :ref:`necessary replication permissions <minio-bucket-replication-serverside-twoway-permissions>`.
You should have two ARNs at the conclusion of this step - one created on ``ALPHA/BUCKET`` pointing at ``BAKER/BUCKET``, and one created on ``BAKER/BUCKET`` pointing at ``ALPHA/BUCKET``. Use the :mc-cmd:`mc admin bucket remote ls` command to verify the created replication remote targets before proceeding. .. _minio-bucket-replication-two-way-minio-cli-create-remote-targets:
2) Create a New Bucket Replication Rule 1) Create Replication Remote Targets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++
Use the :mc-cmd:`mc replicate add` command to add the new server-side .. include:: /includes/common/bucket-replication.rst
replication rule to the each MinIO deployment. :start-after: start-create-replication-remote-targets-cli-desc
:end-before: end-create-replication-remote-targets-cli-desc
.. code-block:: shell Repeat this step on the second MinIO deployment, reversing the origin and destination.
:class: copyable
mc replicate add ALPHA/BUCKET \ You should have two ARNs at the conclusion of this step that point from each deployment to the other deployment's bucket.
--remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \ Use :mc-cmd:`mc admin bucket remote ls` to verify the remote targets before proceeding.
--replicate "delete,delete-marker,existing-objects"
- Replace ``BUCKET`` with the name of the bucket on the ``ALPHA`` deployment to use as the replication source. Replace ``ALPHA`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication. The name *must* match the bucket specified when creating the remote target in the previous step. .. _minio-bucket-replication-two-way-minio-cli-create-replication-rules:
- Replace the ``--remote-bucket`` value with the ARN returned in the previous step. Ensure you specify the ARN created on the ``ALPHA`` deployment. You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment. 2) Create a New Bucket Replication Rule on Each Deployment
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features: .. include:: /includes/common/bucket-replication.rst
:start-after: start-create-bucket-replication-rule-cli-desc
- :ref:`Replication of Deletes <minio-replication-behavior-delete>` :end-before: end-create-bucket-replication-rule-cli-desc
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`. Repeat this step on the other MinIO deployment.
Change the alias for the different origin.
Change the ARN to the ARN generated on the second deployment for the desired bucket.
Repeat this step on the second MinIO deployment, replacing the ``ALPHA`` alias with the ``BAKER`` alias and the ``HOSTNAME`` with the URL of the ``ALPHA`` deployment. You should have two replication rules configured at the conclusion of this step - one created on each deployment that points to the bucket on the other deployment.
Use the :mc-cmd:`mc replicate ls` command to verify the created replication rules.
You should have two replication rules configured at the conclusion of this step - one created on ``ALPHA/BUCKET`` and one created on ``BAKER/BUCKET``. Use the :mc-cmd:`mc replicate ls` command to verify the created replication rules. .. _minio-bucket-replication-two-way-minio-cli-verify-replication-config:
3) Validate the Replication Configuration 3) Validate the Replication Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++++++++++
Use :mc-cmd:`mc cp` to copy a new object to the ``ALPHA/BUCKET`` bucket. .. include:: /includes/common/bucket-replication.rst
:start-after: start-validate-bucket-replication-cli-desc
:end-before: end-validate-bucket-replication-cli-desc
.. code-block:: shell Repeat this test by copying another object to the second deployment and verifying the object replicates to the first deployment.
:class: copyable
mc cp ~/foo.txt ALPHA/BUCKET Once both objects exist on both deployments, you have successfully set up two-way, active-active replication between MinIO buckets.
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket: .. seealso::
.. code-block:: shell - Use the :mc-cmd:`mc replicate edit` command to modify an existing
:class: copyable replication rule.
mc ls BAKER/BUCKET - Use the :mc-cmd:`mc replicate edit` command with the :mc-cmd:`--state "disable" <mc replicate edit --state>` flag to disable an existing replication rule.
Repeat this test by copying a new object to the ``Baker`` source bucket. - Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule.
.. code-block:: shell
:class: copyable
mc cp ~/otherfoo.txt BAKER/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls ALPHA/BUCKET

View File

@ -137,16 +137,14 @@ MinioClient minioClient = new MinioClient()
### AWS S3 ### AWS S3
```cs ```cs
// 1. public MinioClient(String endpoint, String accessKey, String secretKey) // 1. Using Builder with public MinioClient(), Endpoint, Credentials, Secure connection & proxy
MinioClient s3Client = new MinioClient("s3.amazonaws.com", MinioClient s3Client = new MinioClient()
accessKey:"YOUR-ACCESSKEYID", .WithEndpoint("s3.amazonaws.com")
secretKey:"YOUR-SECRETACCESSKEY"); .WithCredentials("YOUR-AWS-ACCESSKEYID", "YOUR-AWS-SECRETACCESSKEY")
// 2. Using Builder with public MinioClient(), Endpoint, Credentials & Secure connection .WithSSL()
MinioClient minioClient = new MinioClient() .WithProxy(proxy)
.WithEndpoint("s3.amazonaws.com") .Build();
.WithCredentials("YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY")
.WithSSL()
.Build()
``` ```
## 2. Bucket operations ## 2. Bucket operations

View File

@ -184,7 +184,7 @@ Glossary
replication replication
mirror mirror
The duplication of a :ref:`bucket <minio-bucket-replication>` or entire :ref:`site <minio-site-replication-overview>` to another location. The replication of a :ref:`bucket <minio-bucket-replication>` or entire :ref:`site <minio-site-replication-overview>` to another location.
scanner scanner
MinIO Scanner MinIO Scanner

File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 96 KiB

After

Width:  |  Height:  |  Size: 80 KiB

File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 57 KiB

After

Width:  |  Height:  |  Size: 56 KiB

File diff suppressed because it is too large Load Diff

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 56 KiB

View File

@ -6,7 +6,9 @@ MinIO supports replication of objects encrypted using :ref:`SSE-KMS <minio-encry
- For objects encrypted using :ref:`SSE-S3 <minio-encryption-sse-s3>`, MinIO *requires* that the target bucket also support SSE-S3 encryption of objects regardless of key name. - For objects encrypted using :ref:`SSE-S3 <minio-encryption-sse-s3>`, MinIO *requires* that the target bucket also support SSE-S3 encryption of objects regardless of key name.
As part of the replication process, MinIO *decrypts* the object on the source bucket and transmits the unencrypted object over the network. The destination MinIO deployment then re-encrypts the object using the encryption settings from the target. MinIO therefore *strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and destination deployments to ensure the safety of objects during transmission. As part of the replication process, MinIO *decrypts* the object on the source bucket and transmits the unencrypted object over the network.
The destination MinIO deployment then re-encrypts the object using the encryption settings from the target.
MinIO therefore *strongly recommends* :ref:`enabling TLS <minio-TLS>` on both source and destination deployments to ensure the safety of objects during transmission.
MinIO does *not* support replicating client-side encrypted objects (SSE-C). MinIO does *not* support replicating client-side encrypted objects (SSE-C).
@ -14,11 +16,10 @@ MinIO does *not* support replicating client-side encrypted objects (SSE-C).
.. start-replication-minio-only .. start-replication-minio-only
MinIO server-side replication only works between MinIO deployments. Both the MinIO server-side replication only works between MinIO deployments.
source and destination deployments *must* run MinIO. Both the source and destination deployments *must* run MinIO.
To configure replication between arbitrary S3-compatible services, To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`.
use :mc-cmd:`mc mirror`.
.. end-replication-minio-only .. end-replication-minio-only
@ -26,15 +27,20 @@ use :mc-cmd:`mc mirror`.
MinIO relies on the immutability protections provided by :ref:`versioning <minio-bucket-versioning>` to support replication and resynchronization. MinIO relies on the immutability protections provided by :ref:`versioning <minio-bucket-versioning>` to support replication and resynchronization.
Use :mc-cmd:`mc version info` to validate the versioning status of both the healthy source and unhealthy target buckets. Use the :mc-cmd:`mc version enable` command to enable versioning as necessary. Use :mc-cmd:`mc version info` to validate the versioning status of both the sourece and remote buckets.
se the :mc-cmd:`mc version enable` command to enable versioning as necessary.
.. end-replication-requires-versioning .. end-replication-requires-versioning
.. start-replication-requires-object-locking .. start-replication-requires-object-locking
MinIO supports replicating objects held under :ref:`WORM Locking <minio-object-locking>`. Both replication buckets *must* have object locking enabled for MinIO to replicate the locked object. For active-active configuration, MinIO recommends using the *same* retention rules on both buckets to ensure consistent behavior across sites. MinIO supports replicating objects held under :ref:`WORM Locking <minio-object-locking>`.
Both replication buckets *must* have object locking enabled for MinIO to replicate the locked object.
For active-active configuration, MinIO recommends using the *same* retention rules on both buckets to ensure consistent behavior across sites.
You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket *prior* to beginning this procedure. You must enable object locking during bucket creation as per S3 behavior.
You can then configure object retention rules at any time.
Configure the necessary rules on the unhealthy target bucket *prior* to beginning this procedure.
.. end-replication-requires-object-locking .. end-replication-requires-object-locking
@ -54,7 +60,9 @@ Bucket replication requires specific permissions on the source and destination d
- The ``"EnableRemoteBucketConfiguration"`` statement grants permission for creating a remote target for supporting replication. - The ``"EnableRemoteBucketConfiguration"`` statement grants permission for creating a remote target for supporting replication.
- The ``"EnableReplicationRuleConfiguration"`` statement grants permission for creating replication rules on a bucket. The ``"arn:aws:s3:::*`` resource applies the replication permissions to *any* bucket on the source deployment. You can restrict the user policy to specific buckets as-needed. - The ``"EnableReplicationRuleConfiguration"`` statement grants permission for creating replication rules on a bucket.
The ``"arn:aws:s3:::*`` resource applies the replication permissions to *any* bucket on the source deployment.
You can restrict the user policy to specific buckets as-needed.
The following code creates a :ref:`MinIO-managed user <minio-users>` with the necessary policy. Replace the ``TARGET`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication: The following code creates a :ref:`MinIO-managed user <minio-users>` with the necessary policy. Replace the ``TARGET`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication:
@ -76,11 +84,14 @@ Bucket replication requires specific permissions on the source and destination d
:class: copyable :class: copyable
:language: json :language: json
- The ``"EnableReplicationOnBucket"`` statement grants permission for a remote target to retrieve bucket-level configuration for supporting replication operations on *all* buckets in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the ``Resource`` array similar to ``"arn:aws:s3:::bucketName"``. - The ``"EnableReplicationOnBucket"`` statement grants permission for a remote target to retrieve bucket-level configuration for supporting replication operations on *all* buckets in the MinIO deployment.
To restrict the policy to specific buckets, specify those buckets as an element in the ``Resource`` array similar to ``"arn:aws:s3:::bucketName"``.
- The ``"EnableReplicatingDataIntoBucket"`` statement grants permission for a remote target to synchronize data into *any* bucket in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the ``Resource`` array similar to ``"arn:aws:s3:::bucketName/*"``. - The ``"EnableReplicatingDataIntoBucket"`` statement grants permission for a remote target to synchronize data into *any* bucket in the MinIO deployment.
To restrict the policy to specific buckets, specify those buckets as an element in the ``Resource`` array similar to ``"arn:aws:s3:::bucketName/*"``.
The following code creates a :ref:`MinIO-managed user <minio-users>` with the necessary policy. Replace the ``TARGET`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication: The following code creates a :ref:`MinIO-managed user <minio-users>` with the necessary policy.
Replace ``TARGET`` with the :ref:`alias <alias>` of the MinIO deployment on which you are configuring replication:
.. code-block:: shell .. code-block:: shell
:class: copyable :class: copyable

View File

@ -0,0 +1,192 @@
.. start-create-replication-remote-targets-cli-desc
Use the :mc-cmd:`mc admin bucket remote add` command to create a replication target from each deployment to the appropriate bucket on the destination deployment.
A bucket may have multiple remote targets to different target buckets.
No two targets can resolve from one bucket to the same remote bucket.
.. code-block:: shell
:class: copyable
mc admin bucket remote add ALIAS/BUCKET \
https://RemoteUser:Password@HOSTNAME/BUCKETDESTINATION \
--service "replication"
- Replace ``ALIAS`` with the :ref:`alias <alias>` of the MinIO deployment that acts as the origin for the replication.
- Replace ``BUCKET`` with the name of the bucket to replicate from on the origin deployment.
- Replacete ``RemoteUser`` with the user name that has the :ref:`necessary replication permissions <minio-bucket-replication-serverside-twoway-permissions>`
- Replace ``Password`` with the secret key for the ``RemoteUser``.
- Replace ``HOSTNAME`` with the URL of the destination deployment.
- Replace ``BUCKETDESTINATION`` with the name of the bucket to replicate to on the destination deployment.
The command returns an :abbr:`ARN <Amazon Resource Name>` similar to the following:
.. code-block:: shell
Role ARN = 'arn:minio:replication::<UUID>:BUCKET'
Copy the ARN to use in the next step, noting the MinIO deployment.
.. end-create-replication-remote-targets-cli-desc
.. start-create-bucket-replication-rule-console-desc
A) Log in to the MinIO Console for the deployment
B) Select the :guilabel:`Manage` button for the bucket to replicate
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: After a successful log in, the MinIO Console shows a list of buckets with options to manage or explore each bucket.
:align: center
C) Select the :guilabel:`Replication` section
.. image:: /images/minio-console/console-iam.png
:width: 600px
:alt: After selecting a bucket to manage, MinIO shows summary information about the bucket as well as a navigation list of pages for adjusting the bucket configuration.
:align: center
D) Select :guilabel:`Add Replication Rule +`
E) Complete the requested information:
.. list-table::
:header-rows: 1
:widths: 25 75
:width: 100%
* - Field
- Description
* - Priority
- Enter a number value to indicate the order in which to process replication rules for the bucket.
`1` indicates the highest importance.
* - Target URL
- The URL of the deployment to replicate data to.
* - Use TLS
- Leave the toggle in the :guilabel:`ON` position if the destination deployment uses TLS.
Otherwise, move the toggle to the :guilabel:`OFF` position.
* - Access Key
- The user name to use on the destination deployment.
The user must have write access to the bucket to replicate to.
* - Secret Key
- The password for the provided **Access Key**.
* - Target Bucket
- The bucket at the destination to write the data to.
The target bucket may have the same name as the origin bucket, depending on the destination bucket location.
* - Region
- The AWS resource region location of the destination deployment.
* - Replication mode
- Leave the default selection of **Asynchronous** to allow MinIO to replicate data after the write operation completes on the origin ment.
Select **Synchronous** to attempt to complete the replication of the object during its write operation.
While synchronous replication may result in more reliable synchronization between the origin and destination buckets, it may also increase the time of each write operation.
* - Bandwidth
- Specify the maximum amount of bandwidth the replication process can use while replicating data.
Enter a number and select a data unit.
* - Health Check Duration
- The maximum length of time in seconds MinIO should spend verifying the health of the replicated data on the destination bucket.
* - Storage Class
- The class of storage to use on the destination deployment for the replicated data.
Valid values are either ``STANDARD`` or ``REDUCED_REDUNDANCY``.
* - Object Filters
- Limit which objects to replicate from the bucket by :term:`Prefix` or **tags**.
If you enter multiple tags, the objects must match all tag values.
* - Metadata Sync
- Leave selected to also replicate the object's metadata file.
Otherwise, move the toggle to the :guilabel:`Off` position.
* - Delete Markers
- Leave selected to also replicate MinIO's indication that an object has been deleted and should also be marked deleted at the ation bucket.
Otherwise, move the toggle to the :guilabel:`Off` position to prevent marking the object as deleted in the destination bucket.
* - Deletes
- Leave selected to allow replication of the deletion of versions of an object.
Otherwise, move the toggle to the :guilabel:`Off` position to not replicate deletion of object versions.
F) Select :guilabel:`Save` to finish adding the replication rule
.. end-create-bucket-replication-rule-console-desc
.. start-create-bucket-replication-rule-cli-desc
Use the :mc-cmd:`mc replicate add` command to add a new replication rule to each MinIO deployment.
.. code-block:: shell
:class: copyable
mc replicate add ALIAS/BUCKET \
--remote-bucket 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
--replicate "delete,delete-marker,existing-objects"
- Replace ``ALIAS`` with the :ref:`alias <alias>` of the origin MinIO deployment.
The name *must* match the bucket specified when creating the remote target in the previous step.
- Replace ``BUCKET`` with the name of the bucket to replicate from on the origin deployment.
- Replace the ``--remote-bucket`` value with the ARN for the destination bucket determined in the first step.
Ensure you specify the ARN created on the origin deployment.
You can use :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured on the deployment.
- The ``--replicate "delete,delete-marker,existing-objects"`` flag enables the following replication features:
- :ref:`Replication of Deletes <minio-replication-behavior-delete>`
- :ref:`Replication of existing Objects <minio-replication-behavior-existing-objects>`
See :mc-cmd:`mc replicate add --replicate` for more complete documentation.
Omit any field to disable replication of that component.
Specify any other supported optional arguments for :mc-cmd:`mc replicate add`.
.. end-create-bucket-replication-rule-cli-desc
.. start-validate-bucket-replication-console-desc
A) Go to the :guilabel:`Buckets` section of the MinIO Console
.. image:: /images/minio-console/console-bucket.png
:width: 600px
:alt: The default screen when logging into the MinIO Console. The screen shows a list of the buckets available in the Deployment with options to Manage or Browse the bucket contents.
:align: center
B) Select the :guilabel:`Browse` button for the bucket you added replication to
.. image:: /images/minio-console/console-object-browser.png
:width: 600px
:alt: The contents of a bucket display after selecting to Browse the MinIO bucket. Options including to Rewind, Refresh, or Upload contents.
C) Select the :guilabel:`Upload` button to add a new object to the bucket
D) Select :guilabel:`Upload File`
E) Use the interface to add a new object to the bucket
F) Go to the other deployment's console and select the destination bucket defined in the replication
.. end-validate-bucket-replication-console-desc
.. start-validate-bucket-replication-cli-desc
Use :mc-cmd:`mc cp` to copy a new object to the replicated bucket on one of the deployments.
.. code-block:: shell
:class: copyable
mc cp ~/foo.txt ALIAS/BUCKET
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
.. code-block:: shell
:class: copyable
mc ls ALIAS/BUCKET
.. end-validate-bucket-replication-cli-desc