mirror of
https://github.com/minio/docs.git
synced 2025-07-27 08:41:57 +03:00
DOCS-330: Continuing Clean Up Work
CR updates
This commit is contained in:
@ -34,6 +34,12 @@ The following table lists MinIO features and their corresponding documentation:
|
|||||||
coding provides object-level healing with less overhead than adjacent
|
coding provides object-level healing with less overhead than adjacent
|
||||||
technologies such as RAID or replication.
|
technologies such as RAID or replication.
|
||||||
|
|
||||||
|
* - :ref:`Server-Side Bucket Replication <minio-bucket-replication-serverside>`
|
||||||
|
- MinIO Server-Side Bucket Replication is an automatic bucket-level
|
||||||
|
configuration that synchronizes objects between MinIO clusters.
|
||||||
|
MinIO supports one-way "active-passive" and two-way "active-active"
|
||||||
|
configurations.
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:titlesonly:
|
:titlesonly:
|
||||||
:hidden:
|
:hidden:
|
||||||
|
2
source/images/active-active-replication.svg
Normal file
2
source/images/active-active-replication.svg
Normal file
File diff suppressed because one or more lines are too long
After Width: | Height: | Size: 22 KiB |
2
source/images/active-passive-replication.svg
Normal file
2
source/images/active-passive-replication.svg
Normal file
File diff suppressed because one or more lines are too long
After Width: | Height: | Size: 16 KiB |
@ -11,32 +11,34 @@ Enable One-Way Server-Side Bucket Replication
|
|||||||
:depth: 2
|
:depth: 2
|
||||||
|
|
||||||
|
|
||||||
MinIO Server-Side Bucket Replication supports automatically synchronizing
|
|
||||||
objects between a source and destination bucket. The destination bucket
|
|
||||||
can be on the same MinIO cluster as the source *or* an independent remote
|
|
||||||
MinIO cluster.
|
|
||||||
|
|
||||||
The procedure on this page creates a new bucket replication rule for
|
The procedure on this page creates a new bucket replication rule for
|
||||||
one-way synchronization of objects from a source MinIO cluster to a destination
|
one-way synchronization of objects between MinIO buckets.
|
||||||
MinIO cluster.
|
|
||||||
|
|
||||||
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
|
.. image:: /images/active-active-replication.svg
|
||||||
replication rule.
|
:width: 600px
|
||||||
|
:alt: Active-Active Replication synchronizes data between two remote clusters.
|
||||||
|
:align: center
|
||||||
|
|
||||||
- Use the :mc-cmd-option:`mc replicate edit` command with the
|
MinIO server-side replication supports at most *two* MinIO clusters. Both
|
||||||
:mc-cmd-option:`--state "disable" <mc replicate edit state>` flag to
|
clusters *must* run MinIO.
|
||||||
disable an existing replication rule.
|
|
||||||
|
|
||||||
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication
|
- To configure replication between arbitrary S3-compatible services, use
|
||||||
rule.
|
:mc-cmd:`mc mirror`.
|
||||||
|
|
||||||
|
- To configure two-way "active-active" replication between MinIO clusters,
|
||||||
|
see :ref:`minio-bucket-replication-serverside-twoway`.
|
||||||
|
|
||||||
.. todo: Diagram
|
.. seealso::
|
||||||
|
|
||||||
.. todo
|
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
|
||||||
This procedure specifically enables only one-way replication between the
|
replication rule.
|
||||||
source and destination buckets. For a procedure on two-way "active-active"
|
|
||||||
replication, see <tutorial>.
|
- Use the :mc-cmd-option:`mc replicate edit` command with the
|
||||||
|
:mc-cmd-option:`--state "disable" <mc replicate edit state>` flag to
|
||||||
|
disable an existing replication rule.
|
||||||
|
|
||||||
|
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication
|
||||||
|
rule.
|
||||||
|
|
||||||
.. _minio-bucket-replication-serverside-oneway-requirements:
|
.. _minio-bucket-replication-serverside-oneway-requirements:
|
||||||
|
|
||||||
@ -72,13 +74,26 @@ Use the :mc-cmd:`mc version enable` command to enable versioning on
|
|||||||
- Replace :mc-cmd:`PATH <mc version enable TARGET>` with the bucket on which
|
- Replace :mc-cmd:`PATH <mc version enable TARGET>` with the bucket on which
|
||||||
to enable versioning.
|
to enable versioning.
|
||||||
|
|
||||||
``mc`` Command Line Interface
|
Install and Configure ``mc`` with Access to Both Clusters.
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
This procedure uses :mc:`mc` for performing operations on both the source
|
This procedure uses :mc:`mc` for performing operations on both the source and
|
||||||
and destination MinIO cluster. See the ``mc``
|
destination MinIO cluster. Install :mc:`mc` on a machine with network access to
|
||||||
:ref:`Installation Quickstart <mc-install>` for instructions on downloading
|
both source and destination clusters. See the ``mc``
|
||||||
and installing ``mc``.
|
:ref:`Installation Quickstart <mc-install>` for instructions on downloading and
|
||||||
|
installing ``mc``.
|
||||||
|
|
||||||
|
Use the :mc:`mc alias` command to create an alias for both MinIO clusters.
|
||||||
|
Alias creation requires specifying an access key for a user on the cluster.
|
||||||
|
This user **must** have permission to create and manage users and policies
|
||||||
|
on the cluster. Specifically, ensure the user has *at minimum*:
|
||||||
|
|
||||||
|
- :policy-action:`admin:CreateUser`
|
||||||
|
- :policy-action:`admin:ListUsers`
|
||||||
|
- :policy-action:`admin:GetUser`
|
||||||
|
- :policy-action:`admin:CreatePolicy`
|
||||||
|
- :policy-action:`admin:GetPolicy`
|
||||||
|
- :policy-action:`admin:AttachUserOrGroupPolicy`
|
||||||
|
|
||||||
.. _minio-bucket-replication-serverside-oneway-permissions:
|
.. _minio-bucket-replication-serverside-oneway-permissions:
|
||||||
|
|
||||||
@ -90,10 +105,10 @@ source and destination clusters:
|
|||||||
|
|
||||||
.. tabs::
|
.. tabs::
|
||||||
|
|
||||||
.. tab:: Source Policy
|
.. tab:: Replication Admin
|
||||||
|
|
||||||
The source cluster *must* have a user with *at minimum* following attached
|
The following policy provides permissions for configuring and enabling
|
||||||
*or* inherited policy:
|
replication on a cluster.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
@ -107,7 +122,7 @@ source and destination clusters:
|
|||||||
"admin:GetBucketTarget"
|
"admin:GetBucketTarget"
|
||||||
],
|
],
|
||||||
"Effect": "Allow",
|
"Effect": "Allow",
|
||||||
"Sid": ""
|
"Sid": "EnableRemoteBucketConfiguration"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"Effect": "Allow",
|
"Effect": "Allow",
|
||||||
@ -116,25 +131,37 @@ source and destination clusters:
|
|||||||
"s3:ListBucket",
|
"s3:ListBucket",
|
||||||
"s3:ListBucketMultipartUploads",
|
"s3:ListBucketMultipartUploads",
|
||||||
"s3:GetBucketLocation",
|
"s3:GetBucketLocation",
|
||||||
"s3:GetBucketVersioning"
|
"s3:GetBucketVersioning",
|
||||||
|
"s3:GetObjectRetention",
|
||||||
|
"s3:GetObjectLegalHold",
|
||||||
|
"s3:PutReplicationConfiguration"
|
||||||
],
|
],
|
||||||
"Resource": [
|
"Resource": [
|
||||||
"arn:aws:s3:::SOURCEBUCKETNAME"
|
"arn:aws:s3:::*"
|
||||||
]
|
],
|
||||||
|
"Sid": "EnableReplicationRuleConfiguration"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
Replace ``SOURCEBUCKETNAME`` with the name of the source bucket from which
|
- The ``"EnableRemoteBucketConfiguration"`` statement grants permission
|
||||||
MinIO replicates objects.
|
for creating a remote target for supporting replication.
|
||||||
|
|
||||||
Use the :mc-cmd:`mc admin policy set` command to associate the policy to
|
- The ``"EnableReplicationRuleConfiguration"`` statement grants permission
|
||||||
a user on the source MinIO cluster.
|
for creating replication rules on a bucket. The ``"arn:aws:s3:::*``
|
||||||
|
resource applies the replication permissions to *any* bucket on the
|
||||||
|
source cluster. You can restrict the user policy to specific buckets
|
||||||
|
as-needed.
|
||||||
|
|
||||||
.. tab:: Destination Policy
|
Use the :mc-cmd:`mc admin policy add` to add this policy to the
|
||||||
|
source cluster. Use :mc-cmd:`mc admin user add` to create a user
|
||||||
|
on the source cluster and :mc-cmd:`mc admin policy set` to associate
|
||||||
|
the policy to that new user.
|
||||||
|
|
||||||
The destination cluster *must* have a user with *at minimum* the
|
.. tab:: Replication Remote User
|
||||||
following attached *or* inherited policy:
|
|
||||||
|
The following policy provides permissions for enabling synchronization of
|
||||||
|
replicated data *into* the cluster.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
@ -150,11 +177,13 @@ source and destination clusters:
|
|||||||
"s3:ListBucketMultipartUploads",
|
"s3:ListBucketMultipartUploads",
|
||||||
"s3:GetBucketLocation",
|
"s3:GetBucketLocation",
|
||||||
"s3:GetBucketVersioning",
|
"s3:GetBucketVersioning",
|
||||||
"s3:GetBucketObjectLockConfiguration"
|
"s3:GetBucketObjectLockConfiguration",
|
||||||
|
"s3:GetEncryptionConfiguration"
|
||||||
],
|
],
|
||||||
"Resource": [
|
"Resource": [
|
||||||
"arn:aws:s3:::DESTINATIONBUCKETNAME"
|
"arn:aws:s3:::*"
|
||||||
]
|
],
|
||||||
|
"Sid": "EnableReplicationOnBucket"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"Effect": "Allow",
|
"Effect": "Allow",
|
||||||
@ -166,23 +195,39 @@ source and destination clusters:
|
|||||||
"s3:GetObjectVersion",
|
"s3:GetObjectVersion",
|
||||||
"s3:GetObjectVersionTagging",
|
"s3:GetObjectVersionTagging",
|
||||||
"s3:PutObject",
|
"s3:PutObject",
|
||||||
|
"s3:PutObjectRetention",
|
||||||
|
"s3:PutBucketObjectLockConfiguration",
|
||||||
|
"s3:PutObjectLegalHold",
|
||||||
"s3:DeleteObject",
|
"s3:DeleteObject",
|
||||||
"s3:ReplicateObject",
|
"s3:ReplicateObject",
|
||||||
"s3:ReplicateDelete"
|
"s3:ReplicateDelete"
|
||||||
],
|
],
|
||||||
"Resource": [
|
"Resource": [
|
||||||
"arn:aws:s3:::DESTINATIONBUCKETNAME/*"
|
"arn:aws:s3:::*"
|
||||||
]
|
],
|
||||||
|
"Sid": "EnableReplicatingDataIntoBucket"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
Replace ``DESTINATIONBUCKETNAME`` with the name of the target bucket to
|
- The ``"EnableReplicationOnBucket"`` statement grants permission for
|
||||||
which MinIO replicates objects.
|
a remote target to retrieve bucket-level configuration for supporting
|
||||||
|
replication operations on *all* buckets in the MinIO cluster. To
|
||||||
|
restrict the policy to specific buckets, specify those buckets as an
|
||||||
|
element in the ``Resource`` array similar to
|
||||||
|
``"arn:aws:s3:::bucketName"``.
|
||||||
|
|
||||||
Use the :mc-cmd:`mc admin policy set` command to associate the policy
|
- The ``"EnableReplicatingDataIntoBucket"`` statement grants permission
|
||||||
to a user on the target MinIO cluster.
|
for a remote target to synchronize data into *any* bucket in the MinIO
|
||||||
|
cluster. To restrict the policy to specific buckets, specify those
|
||||||
|
buckets as an element in the ``Resource`` array similar to
|
||||||
|
``"arn:aws:s3:::bucketName/*"``.
|
||||||
|
|
||||||
|
Use the :mc-cmd:`mc admin policy add` to add this policy to the
|
||||||
|
destination cluster. Use :mc-cmd:`mc admin user add` to create a user
|
||||||
|
on the destination cluster and :mc-cmd:`mc admin policy set` to associate
|
||||||
|
the policy to that new user.
|
||||||
|
|
||||||
MinIO strongly recommends creating users specifically for supporting
|
MinIO strongly recommends creating users specifically for supporting
|
||||||
bucket replication operations. See
|
bucket replication operations. See
|
||||||
:mc:`mc admin user` and :mc:`mc admin policy` for more complete
|
:mc:`mc admin user` and :mc:`mc admin policy` for more complete
|
||||||
@ -199,6 +244,14 @@ does *not* apply replication rules to existing objects in the bucket. Use
|
|||||||
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
:mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination
|
||||||
bucket.
|
bucket.
|
||||||
|
|
||||||
|
For buckets with active write operations during the procedure, any objects
|
||||||
|
written *before* configuring bucket replication remain unreplicated.
|
||||||
|
|
||||||
|
Consider scheduling a maintenance period during which applications stop
|
||||||
|
all write operations to the bucket or buckets for which you are configuring
|
||||||
|
bucket replication. Restart write operations at the completion of the
|
||||||
|
procedure to ensure consistent object replication.
|
||||||
|
|
||||||
Replication of Delete Operations
|
Replication of Delete Operations
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
@ -237,51 +290,66 @@ MinIO does *not* support replicating client-side encrypted objects
|
|||||||
Procedure
|
Procedure
|
||||||
---------
|
---------
|
||||||
|
|
||||||
1) Configure ``mc`` Access to the Source and Destination Cluster
|
1) Configure User Accounts and Policies for Replication
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This step creates users and policies on both MinIO clusters for
|
||||||
|
supporting replication operations. You can skip this step if both
|
||||||
|
clusters already have users with the necessary
|
||||||
|
:ref:`permissions <minio-bucket-replication-serverside-oneway-permissions>`.
|
||||||
|
|
||||||
|
The following examples use ``Alpha`` and ``Baker`` as placeholder :mc:`aliases
|
||||||
|
<mc alias>` for each MinIO cluster. You should replace these values with the
|
||||||
|
appropriate aliases for the MinIO clusters on which you are configuring bucket
|
||||||
|
replication. These examples assume that the specified aliases have
|
||||||
|
the necessary permissions for creating policies and users on both clusters. See
|
||||||
|
:ref:`minio-users` and :ref:`minio-policy` for more complete documentation on
|
||||||
|
MinIO users and policies respectively.
|
||||||
|
|
||||||
|
A\) Create Replication Administrator
|
||||||
|
The following code creates a user and policy for supporting configuring
|
||||||
|
replication on the ``Alpha`` cluster. Replace the
|
||||||
|
password ``LongRandomSecretKey`` with a long, random, and secure secret key
|
||||||
|
as per your organizations best practices for password generation.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationAdminPolicy.json | \
|
||||||
|
mc admin policy add Alpha ReplicationAdminPolicy /dev/stdin
|
||||||
|
mc admin user add Alpha alphaReplicationAdmin LongRandomSecretKey
|
||||||
|
mc admin policy set Alpha ReplicationAdminPolicy user=alphaReplicationAdmin
|
||||||
|
|
||||||
|
B\) Create Remote Replication User
|
||||||
|
The following code creates a user and policy for supporting synchronizing
|
||||||
|
data into the ``Baker`` cluster. Replace the password
|
||||||
|
``LongRandomSecretKey`` with a long, random, and secure secret key as per
|
||||||
|
your organizations best practices for password generation.
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
|
wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationRemoteUserPolicy.json | \
|
||||||
|
mc admin policy add Baker ReplicationRemoteUserPolicy /dev/stdin
|
||||||
|
mc admin user add Baker bakerReplicationRemoteUser LongRandomSecretKey
|
||||||
|
mc admin policy set Baker ReplicationRemoteUserPolicy user=bakerReplicationRemoteUser
|
||||||
|
|
||||||
|
2) Configure ``mc`` Access to the Source and Destination Cluster
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Use the :mc-cmd:`mc alias set` command to add an alias for both source
|
Use the :mc-cmd:`mc alias set` command to add an alias for both source
|
||||||
and destination MinIO clusters.
|
and destination MinIO clusters.
|
||||||
|
|
||||||
.. tabs::
|
Use the :mc-cmd:`mc alias set` command to add a replication-specific alias for
|
||||||
|
both remote clusters:
|
||||||
|
|
||||||
.. tab: Source Cluster
|
.. code-block:: shell
|
||||||
|
:class: copyable
|
||||||
|
|
||||||
.. code-block:: shell
|
mc admin set AlphaReplication HOSTNAME AlphaReplicationAdmin LongRandomSecretKey
|
||||||
:class: copyable
|
mc admin set BakerReplication HOSTNAME BakerReplicationUser LongRandomSecretKey
|
||||||
|
|
||||||
mc alias set SourceCluster HOSTNAME ACCESSKEY SECRETKEY
|
3) Create a Replication Target for the Destination Cluster
|
||||||
|
|
||||||
- Replace :mc-cmd:`~mc alias set HOSTNAME` with the hostname or IP address
|
|
||||||
of a node in the source MinIO cluster. For distributed MinIO clusters
|
|
||||||
using a load balancer, specify the hostname or IP address of that load
|
|
||||||
balancer.
|
|
||||||
|
|
||||||
- Replace :mc-cmd:`~mc alias set ACCESSKEY` and
|
|
||||||
:mc-cmd:`~mc alias set SECRETKEY` with the access and secret key for a
|
|
||||||
user with the :ref:`required permissions
|
|
||||||
<minio-bucket-replication-serverside-oneway-permissions>` on the source
|
|
||||||
MinIO cluster.
|
|
||||||
|
|
||||||
.. tab:: DestinationCluster
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
:class: copyable
|
|
||||||
|
|
||||||
mc alias set DestinationCluster HOSTNAME ACCESSKEY SECRETKEY
|
|
||||||
|
|
||||||
- Replace :mc-cmd:`~mc alias set HOSTNAME` with the hostname or IP address
|
|
||||||
of a node in the destination MinIO cluster. For distributed MinIO
|
|
||||||
clusters using a load balancer, specify the hostname or IP address of
|
|
||||||
that load balancer.
|
|
||||||
|
|
||||||
- Replace :mc-cmd:`~mc alias set ACCESSKEY` and
|
|
||||||
:mc-cmd:`~mc alias set SECRETKEY` with the access and secret key for a
|
|
||||||
user with the :ref:`required permissions
|
|
||||||
<minio-bucket-replication-serverside-oneway-permissions>` on the
|
|
||||||
destination MinIO cluster.
|
|
||||||
|
|
||||||
2) Create a Replication Target for the Destination Cluster
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Use the :mc-cmd:`mc admin bucket remote` command to create a replication target
|
Use the :mc-cmd:`mc admin bucket remote` command to create a replication target
|
||||||
@ -292,35 +360,24 @@ bucket.
|
|||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
|
|
||||||
mc admin bucket remote add SourceCluster/SOURCEBUCKET \
|
mc admin bucket remote add AlphaReplication/SOURCEBUCKET \
|
||||||
https://ACCESSKEY:SECRETKEY@HOSTNAME/DESTINATIONBUCKET
|
https://bakerReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/DESTINATIONBUCKET
|
||||||
--service "replication"
|
--service "replication"
|
||||||
[--sync]
|
[--sync]
|
||||||
|
|
||||||
- Replace ``SOURCEBUCKET`` with the name of the bucket on the source cluster.
|
- Replace ``SOURCEBUCKET`` with the name of the source bucket on the
|
||||||
The name of the bucket *must* match the bucket name specified to the source
|
``Alpha`` cluster.
|
||||||
user's replication :ref:`policy
|
|
||||||
<minio-bucket-replication-serverside-oneway-permissions>`.
|
|
||||||
|
|
||||||
- Replace ``ACCESSKEY`` and ``SECRETKEY`` with the access and secret key for
|
- Replace ``HOSTNAME`` with the URL of the ``Baker`` cluster.
|
||||||
a user with the :ref:`required permissions
|
|
||||||
<minio-bucket-replication-serverside-oneway-permissions>` on the destination
|
|
||||||
MinIO cluster.
|
|
||||||
|
|
||||||
- Replace the ``HOSTNAME`` with the hostname or IP address of
|
- Replace ``DESTINATIONBUCKET`` with the name of the target bucket on the
|
||||||
a node in the MinIO cluster. For distributed MinIO clusters using a
|
``Baker`` cluster.
|
||||||
load balancer, specify the hostname or IP address of that load balancer.
|
|
||||||
|
|
||||||
- Replace the ``DESTINATIONBUCKET`` with the name of the bucket on the
|
- Specify the :mc-cmd-option:`~mc admin bucket remote add sync` option to
|
||||||
destination cluster. The name of the bucket *must* match the bucket name
|
enable synchronous replication. Omit the option to use the default of
|
||||||
specified to the destination user's replication :ref:`policy
|
asynchronous replication. See the reference documentation for
|
||||||
<minio-bucket-replication-serverside-oneway-permissions>`.
|
:mc-cmd-option:`~mc admin bucket remote add sync` for more information
|
||||||
|
on synchronous vs asynchronous replication.
|
||||||
- MinIO defaults to using asynchronous object replication, where MinIO
|
|
||||||
replicates objects *after* returning the PUT object response. Specify the
|
|
||||||
:mc-cmd-option:`~mc admin bucket remote add sync` option to enable
|
|
||||||
synchronous replication, where MinIO attempts to replicate the object
|
|
||||||
*prior* to returning the PUT object response.
|
|
||||||
|
|
||||||
The command returns an ARN similar to the following:
|
The command returns an ARN similar to the following:
|
||||||
|
|
||||||
@ -330,7 +387,7 @@ The command returns an ARN similar to the following:
|
|||||||
|
|
||||||
Copy the ARN string for use in the next step.
|
Copy the ARN string for use in the next step.
|
||||||
|
|
||||||
3) Create a New Bucket Replication Rule
|
4) Create a New Bucket Replication Rule
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Use the :mc-cmd:`mc replicate add` command to add the new server-side
|
Use the :mc-cmd:`mc replicate add` command to add the new server-side
|
||||||
@ -339,29 +396,35 @@ replication rule to the source MinIO cluster.
|
|||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
|
|
||||||
mc replicate add SourceCluster/SOURCEBUCKET \
|
mc replicate add AlphaReplication/SOURCEBUCKET \
|
||||||
--remote-bucket DESTINATIONBUCKET \
|
--remote-bucket DESTINATIONBUCKET \
|
||||||
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
|
||||||
--replicate "delete,delete-marker"
|
--replicate "delete,delete-marker"
|
||||||
|
|
||||||
- Replace ``SOURCEBUCKET`` with the name of the bucket on the source cluster.
|
- Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha
|
||||||
|
replicates data. The name *must* match the bucket specified when
|
||||||
|
creating the remote target in the previous step.
|
||||||
|
|
||||||
- Replace the ``DESTINATIONBUCKET`` with the name of the bucket on the
|
- Replace the ``DESTINATIONBUCKET`` with the name of the ``Baker`` bucket to
|
||||||
destination cluster.
|
which Alpha replicates data. The name *must* match the bucket specified
|
||||||
|
when creating the remote target in the previous step.
|
||||||
|
|
||||||
- Replace the ``--arn`` value with the ARN returned in the previous step.
|
- Replace the ``--arn`` value with the ARN returned in the previous step.
|
||||||
|
Ensure you specify the ARN created on the ``Alpha`` cluster. You can use
|
||||||
- Replace ``PRIORITY`` with the integer priority of this replication rule.
|
:mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured
|
||||||
|
on the cluster.
|
||||||
|
|
||||||
|
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
||||||
|
markers and deletion of object versions. See
|
||||||
|
:mc-cmd-option:`mc replicate add replicate` for more complete
|
||||||
|
documentation. Omit these fields to disable replication of delete
|
||||||
|
operations.
|
||||||
|
|
||||||
- The ``--replicate "delete,delete-marker"`` flag enables replicating delete
|
|
||||||
markers and deletion of object versions. See
|
|
||||||
:mc-cmd-option:`mc replicate add replicate` for more complete documentation.
|
|
||||||
Omit this field to disable replication of delete operations.
|
|
||||||
|
|
||||||
Specify any other supported optional arguments for
|
Specify any other supported optional arguments for
|
||||||
:mc-cmd:`mc replicate add`.
|
:mc-cmd:`mc replicate add`.
|
||||||
|
|
||||||
4) Validate the Replication Configuration
|
5) Validate the Replication Configuration
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
|
Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
|
||||||
@ -369,14 +432,14 @@ Use :mc-cmd:`mc cp` to copy a new object to the source bucket.
|
|||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
|
|
||||||
mc cp ~/foo.txt SourceCluster/SOURCEBUCKET
|
mc cp ~/foo.txt Alpha/SOURCEBUCKET
|
||||||
|
|
||||||
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
|
Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code-block:: shell
|
||||||
:class: copyable
|
:class: copyable
|
||||||
|
|
||||||
mc ls DestinationCluster/DESTINATIONBUCKET
|
mc ls Baker/DESTINATIONBUCKET
|
||||||
|
|
||||||
If the remote target was configured *without* the
|
If the remote target was configured *without* the
|
||||||
:mc-cmd-option:`~mc admin bucket remote add sync` option, the destination
|
:mc-cmd-option:`~mc admin bucket remote add sync` option, the destination
|
||||||
|
@ -14,11 +14,13 @@ Enable Two-Way Server-Side Bucket Replication
|
|||||||
The procedure on this page creates a new bucket replication rule for two-way
|
The procedure on this page creates a new bucket replication rule for two-way
|
||||||
"active-active" synchronization of objects between MinIO buckets.
|
"active-active" synchronization of objects between MinIO buckets.
|
||||||
|
|
||||||
.. todo: Diagram
|
.. image:: /images/active-active-replication.svg
|
||||||
|
:width: 600px
|
||||||
|
:alt: Active-Active Replication synchronizes data between two remote clusters.
|
||||||
|
:align: center
|
||||||
|
|
||||||
MinIO server-side replication supports active-active replication between
|
MinIO server-side replication supports at most *two* MinIO clusters. Both
|
||||||
at most *two* MinIO clusters. MinIO does not support arbitrary S3-compatible
|
clusters *must* run MinIO.
|
||||||
services.
|
|
||||||
|
|
||||||
- To configure replication between arbitrary S3-compatible services, use
|
- To configure replication between arbitrary S3-compatible services, use
|
||||||
:mc-cmd:`mc mirror`.
|
:mc-cmd:`mc mirror`.
|
||||||
@ -26,6 +28,13 @@ services.
|
|||||||
- To configure one-way "active-passive" replication between MinIO clusters,
|
- To configure one-way "active-passive" replication between MinIO clusters,
|
||||||
see :ref:`minio-bucket-replication-serverside-oneway`.
|
see :ref:`minio-bucket-replication-serverside-oneway`.
|
||||||
|
|
||||||
|
MinIO Active-Active replication is designed for synchronizing objects between
|
||||||
|
two MinIO clusters. MinIO does not support Active-Active replication between
|
||||||
|
more than two clusters (multi-site). Enterprises looking to implement multi-site
|
||||||
|
replication should consider leveraging `MinIO SUBNET
|
||||||
|
<https://min.io/pricing?ref=docs>`__ support to access the expertise, planning,
|
||||||
|
and engineering resources required for addressing that use case.
|
||||||
|
|
||||||
.. seealso::
|
.. seealso::
|
||||||
|
|
||||||
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
|
- Use the :mc-cmd:`mc replicate edit` command to modify an existing
|
||||||
@ -38,13 +47,6 @@ services.
|
|||||||
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication
|
- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication
|
||||||
rule.
|
rule.
|
||||||
|
|
||||||
.. todo: Diagram
|
|
||||||
|
|
||||||
.. todo
|
|
||||||
This procedure specifically enables only two-way replication between the
|
|
||||||
source and destination buckets. For a procedure on two-way "active-active"
|
|
||||||
replication, see <tutorial>.
|
|
||||||
|
|
||||||
.. _minio-bucket-replication-serverside-twoway-requirements:
|
.. _minio-bucket-replication-serverside-twoway-requirements:
|
||||||
|
|
||||||
Requirements
|
Requirements
|
||||||
@ -193,6 +195,7 @@ destination clusters to configure and enable replication rules.
|
|||||||
"s3:GetObjectVersionTagging",
|
"s3:GetObjectVersionTagging",
|
||||||
"s3:PutObject",
|
"s3:PutObject",
|
||||||
"s3:PutObjectRetention",
|
"s3:PutObjectRetention",
|
||||||
|
"s3:PutBucketObjectLockConfiguration",
|
||||||
"s3:PutObjectLegalHold",
|
"s3:PutObjectLegalHold",
|
||||||
"s3:DeleteObject",
|
"s3:DeleteObject",
|
||||||
"s3:ReplicateObject",
|
"s3:ReplicateObject",
|
||||||
@ -291,8 +294,6 @@ MinIO supports replicating objects with retention settings, such as
|
|||||||
source and destination bucket *must* have object locking enabled for MinIO
|
source and destination bucket *must* have object locking enabled for MinIO
|
||||||
to replicate objects with their associated retention settings.
|
to replicate objects with their associated retention settings.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Procedure
|
Procedure
|
||||||
---------
|
---------
|
||||||
|
|
||||||
@ -304,14 +305,13 @@ supporting replication operations. You can skip this step if both
|
|||||||
clusters already have users with the necessary
|
clusters already have users with the necessary
|
||||||
:ref:`permissions <minio-bucket-replication-serverside-twoway-permissions>`.
|
:ref:`permissions <minio-bucket-replication-serverside-twoway-permissions>`.
|
||||||
|
|
||||||
The following examples that the :mc:`alias <mc alias>` for each cluster
|
The following examples use ``Alpha`` and ``Baker`` as placeholder :mc:`aliases
|
||||||
provides the necessary permissions for creating policies and users on both
|
<mc alias>` for each MinIO cluster. You should replace these values with the
|
||||||
clusters. See :ref:`minio-users` and :ref:`minio-policy` for more complete
|
appropriate aliases for the MinIO clusters on which you are configuring bucket
|
||||||
documentation on MinIO users and policies respectively.
|
replication. These examples assume that the specified aliases have
|
||||||
|
the necessary permissions for creating policies and users on both clusters. See
|
||||||
Replace ``Alpha`` and ``Baker`` in the following examples with the appropriate
|
:ref:`minio-users` and :ref:`minio-policy` for more complete documentation on
|
||||||
:mc:`aliases <mc alias>` for the MinIO clusters on which you are configuring
|
MinIO users and policies respectively.
|
||||||
bucket replication.
|
|
||||||
|
|
||||||
A\) Create Replication Administrators
|
A\) Create Replication Administrators
|
||||||
The following code creates policies and users for supporting configuring
|
The following code creates policies and users for supporting configuring
|
||||||
@ -332,7 +332,7 @@ A\) Create Replication Administrators
|
|||||||
mc admin user add Baker bakerReplicationAdmin LongRandomSecretKey
|
mc admin user add Baker bakerReplicationAdmin LongRandomSecretKey
|
||||||
mc admin policy set baker ReplicationAdminPolicy user=bakerReplicationAdmin
|
mc admin policy set baker ReplicationAdminPolicy user=bakerReplicationAdmin
|
||||||
|
|
||||||
B\) Create Remote Replication Administrators
|
B\) Create Remote Replication User
|
||||||
The following code creates policies and users for supporting synchronizing data
|
The following code creates policies and users for supporting synchronizing data
|
||||||
to the ``Alpha`` and ``Baker`` clusters. Replace the password
|
to the ``Alpha`` and ``Baker`` clusters. Replace the password
|
||||||
``LongRandomSecretKey`` with a long, random, and secure secret key as per your
|
``LongRandomSecretKey`` with a long, random, and secure secret key as per your
|
||||||
|
@ -8,7 +8,7 @@ Bucket Replication
|
|||||||
|
|
||||||
.. contents:: Table of Contents
|
.. contents:: Table of Contents
|
||||||
:local:
|
:local:
|
||||||
:depth: 1
|
:depth: 2
|
||||||
|
|
||||||
MinIO supports server-side and client-side replication of objects between source
|
MinIO supports server-side and client-side replication of objects between source
|
||||||
and destination buckets. MinIO offers both active-passive (one-way) and
|
and destination buckets. MinIO offers both active-passive (one-way) and
|
||||||
@ -54,8 +54,6 @@ replication while adding the following MinIO-only features:
|
|||||||
- Source and destination bucket names can match, supporting site-to-site
|
- Source and destination bucket names can match, supporting site-to-site
|
||||||
use cases such as Splunk or Veeam BC/DR.
|
use cases such as Splunk or Veeam BC/DR.
|
||||||
|
|
||||||
- Replicate object locking and retention settings from source to destination.
|
|
||||||
|
|
||||||
- Simplified implementation than S3 bucket replication configuration, removing
|
- Simplified implementation than S3 bucket replication configuration, removing
|
||||||
the need to configure settings like AccessControlTranslation, Metrics, and
|
the need to configure settings like AccessControlTranslation, Metrics, and
|
||||||
SourceSelectionCriteria.
|
SourceSelectionCriteria.
|
||||||
@ -63,6 +61,111 @@ replication while adding the following MinIO-only features:
|
|||||||
- Active-Active (Two-Way) replication of objects between source and destination
|
- Active-Active (Two-Way) replication of objects between source and destination
|
||||||
buckets.
|
buckets.
|
||||||
|
|
||||||
|
.. _minio-replication-process:
|
||||||
|
|
||||||
|
Replication Process
|
||||||
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
MinIO uses a replication queuing system with multiple concurrent replication
|
||||||
|
workers operating on that queue. MinIO continuously works to replicate and
|
||||||
|
remove objects from the queue while scanning for new unreplicated objects to
|
||||||
|
add to the queue.
|
||||||
|
|
||||||
|
MinIO sets the ``X-Amz-Replication-Status`` metadata field according to the
|
||||||
|
replication state of the object:
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:header-rows: 1
|
||||||
|
:width: 100%
|
||||||
|
:widths: 30 70
|
||||||
|
|
||||||
|
* - Replication State
|
||||||
|
- Description
|
||||||
|
|
||||||
|
* - ``PENDING``
|
||||||
|
- The object has not yet been replicated. MinIO applies this state
|
||||||
|
if the object meets one of the configured replication rules on the
|
||||||
|
bucket. MinIO continuously scans for ``PENDING`` objects not yet in the
|
||||||
|
replication queue and adds them to the queue as space is available.
|
||||||
|
|
||||||
|
* - ``COMPLETED``
|
||||||
|
- The object has successfully replicated to the remote cluster.
|
||||||
|
|
||||||
|
* - ``FAILED``
|
||||||
|
- The object failed to replicate to the remote cluster.
|
||||||
|
|
||||||
|
MinIO continuously scans for ``FAILED`` objects not yet in the
|
||||||
|
replication queue and adds them to the queue as space is available.
|
||||||
|
|
||||||
|
* - ``REPLICA``
|
||||||
|
- The object is itself a replica from a remote source.
|
||||||
|
|
||||||
|
MinIO ignores ``REPLICA`` objects, including any metadata-only changes
|
||||||
|
to that object. MinIO does not support
|
||||||
|
AWS `replica modification sync
|
||||||
|
<https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-replication-adds-support-two-way-replication/>`__
|
||||||
|
at this time.
|
||||||
|
|
||||||
|
The replication process generally has one of the following flows:
|
||||||
|
|
||||||
|
- ``PENDING -> COMPLETED``
|
||||||
|
- ``PENDING -> FAILED -> COMPLETED``
|
||||||
|
|
||||||
|
Synchronous vs Asynchronous Replication
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
MinIO supports specifying either asynchronous (default) or synchronous
|
||||||
|
replication for a given remote target.
|
||||||
|
|
||||||
|
With the default asynchronous replication, MinIO completes the originating
|
||||||
|
``PUT`` operation *before* placing the object into a :ref:`replication queue
|
||||||
|
<minio-replication-process>`. The originating client may therefore see a
|
||||||
|
successful ``PUT`` operation *before* the object is replicated. While
|
||||||
|
this may result in stale or missing objects on the remote, it mitigates
|
||||||
|
the risk of slow write operations due to replication load.
|
||||||
|
|
||||||
|
With synchronous replication, MinIO attempts to replicate the object *prior* to
|
||||||
|
completing the originating ``PUT`` operation. MinIO returns a successful ``PUT``
|
||||||
|
operation whether or not the replication attempts succeeds. While this may
|
||||||
|
result in more reliable synchronization between the source and remote target,
|
||||||
|
it may also increase the time of each write operation due to replication load.
|
||||||
|
|
||||||
|
You must explicitly enable synchronous replication when configuring the remote
|
||||||
|
target target using the :mc-cmd:`mc admin bucket remote add` command with the
|
||||||
|
:mc-cmd-option:`~mc admin bucket remote add sync` flag.
|
||||||
|
|
||||||
|
Replication of Delete Operations
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
MinIO supports replicating delete operations, where MinIO synchronizes
|
||||||
|
deleting specific object versions *and* new :s3-docs:`delete markers
|
||||||
|
<delete-marker-replication.html>`. Delete operation replication uses
|
||||||
|
the same :ref:`replication process <minio-replication-process>` as all other
|
||||||
|
replication operations.
|
||||||
|
|
||||||
|
For delete marker replication, MinIO begins the replication process after
|
||||||
|
a delete operation creates the delete marker. MinIO uses the
|
||||||
|
``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking
|
||||||
|
delete marker replication status. In
|
||||||
|
:ref:`active-active <minio-bucket-replication-serverside-twoway>`
|
||||||
|
replication configurations, MinIO may produce duplicate delete markers if
|
||||||
|
both clusters concurrently create a delete marker for an object *or*
|
||||||
|
if one or both clusters were down before the replication event synchronized.
|
||||||
|
|
||||||
|
For replicating the deletion of a specific object version, MinIO marks the
|
||||||
|
object version as ``PENDING`` until replication completes. Once the remote
|
||||||
|
target deletes that object version, MinIO deletes the object on the source.
|
||||||
|
While this process ensures near-synchronized version deletion, it may result
|
||||||
|
in listing operations returning the object version after the initial
|
||||||
|
delete operation. MinIO uses the ``X-Minio-Replication-Delete-Status`` for
|
||||||
|
tracking delete version replication status.
|
||||||
|
|
||||||
|
MinIO requires explicitly enabling versioned deletes and delete marker
|
||||||
|
replication . Use the :mc-cmd-option:`mc replicate add replicate` field to
|
||||||
|
specify both or either ``delete`` and ``delete-marker`` to enable versioned
|
||||||
|
deletes and delete marker replication respectively. To enable both, specify both
|
||||||
|
strings using a comma separator ``delete,delete-marker``.
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:hidden:
|
:hidden:
|
||||||
:titlesonly:
|
:titlesonly:
|
||||||
|
Reference in New Issue
Block a user