From ae77ac4740a24b829a1eb1777dd6bc0d17975672 Mon Sep 17 00:00:00 2001 From: ravindk89 Date: Mon, 12 Apr 2021 16:55:22 -0400 Subject: [PATCH] DOCS-330: Continuing Clean Up Work CR updates --- source/concepts/feature-overview.rst | 6 + source/images/active-active-replication.svg | 2 + source/images/active-passive-replication.svg | 2 + ...server-side-one-way-bucket-replication.rst | 309 +++++++++++------- ...server-side-two-way-bucket-replication.rst | 44 +-- source/replication/replication-overview.rst | 109 +++++- 6 files changed, 324 insertions(+), 148 deletions(-) create mode 100644 source/images/active-active-replication.svg create mode 100644 source/images/active-passive-replication.svg diff --git a/source/concepts/feature-overview.rst b/source/concepts/feature-overview.rst index cea8008c..fd65b74c 100644 --- a/source/concepts/feature-overview.rst +++ b/source/concepts/feature-overview.rst @@ -34,6 +34,12 @@ The following table lists MinIO features and their corresponding documentation: coding provides object-level healing with less overhead than adjacent technologies such as RAID or replication. + * - :ref:`Server-Side Bucket Replication ` + - MinIO Server-Side Bucket Replication is an automatic bucket-level + configuration that synchronizes objects between MinIO clusters. + MinIO supports one-way "active-passive" and two-way "active-active" + configurations. + .. toctree:: :titlesonly: :hidden: diff --git a/source/images/active-active-replication.svg b/source/images/active-active-replication.svg new file mode 100644 index 00000000..f3f29117 --- /dev/null +++ b/source/images/active-active-replication.svg @@ -0,0 +1,2 @@ + + diff --git a/source/images/active-passive-replication.svg b/source/images/active-passive-replication.svg new file mode 100644 index 00000000..e3e621f3 --- /dev/null +++ b/source/images/active-passive-replication.svg @@ -0,0 +1,2 @@ + + diff --git a/source/replication/enable-server-side-one-way-bucket-replication.rst b/source/replication/enable-server-side-one-way-bucket-replication.rst index c01fd92e..b1516ea1 100644 --- a/source/replication/enable-server-side-one-way-bucket-replication.rst +++ b/source/replication/enable-server-side-one-way-bucket-replication.rst @@ -11,32 +11,34 @@ Enable One-Way Server-Side Bucket Replication :depth: 2 -MinIO Server-Side Bucket Replication supports automatically synchronizing -objects between a source and destination bucket. The destination bucket -can be on the same MinIO cluster as the source *or* an independent remote -MinIO cluster. - The procedure on this page creates a new bucket replication rule for -one-way synchronization of objects from a source MinIO cluster to a destination -MinIO cluster. +one-way synchronization of objects between MinIO buckets. -- Use the :mc-cmd:`mc replicate edit` command to modify an existing - replication rule. +.. image:: /images/active-active-replication.svg + :width: 600px + :alt: Active-Active Replication synchronizes data between two remote clusters. + :align: center -- Use the :mc-cmd-option:`mc replicate edit` command with the - :mc-cmd-option:`--state "disable" ` flag to - disable an existing replication rule. +MinIO server-side replication supports at most *two* MinIO clusters. Both +clusters *must* run MinIO. -- Use the :mc-cmd:`mc replicate rm` command to remove an existing replication - rule. +- To configure replication between arbitrary S3-compatible services, use + :mc-cmd:`mc mirror`. +- To configure two-way "active-active" replication between MinIO clusters, + see :ref:`minio-bucket-replication-serverside-twoway`. -.. todo: Diagram +.. seealso:: -.. todo - This procedure specifically enables only one-way replication between the - source and destination buckets. For a procedure on two-way "active-active" - replication, see . + - Use the :mc-cmd:`mc replicate edit` command to modify an existing + replication rule. + + - Use the :mc-cmd-option:`mc replicate edit` command with the + :mc-cmd-option:`--state "disable" ` flag to + disable an existing replication rule. + + - Use the :mc-cmd:`mc replicate rm` command to remove an existing replication + rule. .. _minio-bucket-replication-serverside-oneway-requirements: @@ -72,13 +74,26 @@ Use the :mc-cmd:`mc version enable` command to enable versioning on - Replace :mc-cmd:`PATH ` with the bucket on which to enable versioning. -``mc`` Command Line Interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Install and Configure ``mc`` with Access to Both Clusters. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This procedure uses :mc:`mc` for performing operations on both the source -and destination MinIO cluster. See the ``mc`` -:ref:`Installation Quickstart ` for instructions on downloading -and installing ``mc``. +This procedure uses :mc:`mc` for performing operations on both the source and +destination MinIO cluster. Install :mc:`mc` on a machine with network access to +both source and destination clusters. See the ``mc`` +:ref:`Installation Quickstart ` for instructions on downloading and +installing ``mc``. + +Use the :mc:`mc alias` command to create an alias for both MinIO clusters. +Alias creation requires specifying an access key for a user on the cluster. +This user **must** have permission to create and manage users and policies +on the cluster. Specifically, ensure the user has *at minimum*: + +- :policy-action:`admin:CreateUser` +- :policy-action:`admin:ListUsers` +- :policy-action:`admin:GetUser` +- :policy-action:`admin:CreatePolicy` +- :policy-action:`admin:GetPolicy` +- :policy-action:`admin:AttachUserOrGroupPolicy` .. _minio-bucket-replication-serverside-oneway-permissions: @@ -90,10 +105,10 @@ source and destination clusters: .. tabs:: - .. tab:: Source Policy + .. tab:: Replication Admin - The source cluster *must* have a user with *at minimum* following attached - *or* inherited policy: + The following policy provides permissions for configuring and enabling + replication on a cluster. .. code-block:: shell :class: copyable @@ -107,7 +122,7 @@ source and destination clusters: "admin:GetBucketTarget" ], "Effect": "Allow", - "Sid": "" + "Sid": "EnableRemoteBucketConfiguration" }, { "Effect": "Allow", @@ -116,25 +131,37 @@ source and destination clusters: "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:GetBucketLocation", - "s3:GetBucketVersioning" + "s3:GetBucketVersioning", + "s3:GetObjectRetention", + "s3:GetObjectLegalHold", + "s3:PutReplicationConfiguration" ], "Resource": [ - "arn:aws:s3:::SOURCEBUCKETNAME" - ] + "arn:aws:s3:::*" + ], + "Sid": "EnableReplicationRuleConfiguration" } ] } - Replace ``SOURCEBUCKETNAME`` with the name of the source bucket from which - MinIO replicates objects. + - The ``"EnableRemoteBucketConfiguration"`` statement grants permission + for creating a remote target for supporting replication. - Use the :mc-cmd:`mc admin policy set` command to associate the policy to - a user on the source MinIO cluster. + - The ``"EnableReplicationRuleConfiguration"`` statement grants permission + for creating replication rules on a bucket. The ``"arn:aws:s3:::*`` + resource applies the replication permissions to *any* bucket on the + source cluster. You can restrict the user policy to specific buckets + as-needed. - .. tab:: Destination Policy + Use the :mc-cmd:`mc admin policy add` to add this policy to the + source cluster. Use :mc-cmd:`mc admin user add` to create a user + on the source cluster and :mc-cmd:`mc admin policy set` to associate + the policy to that new user. - The destination cluster *must* have a user with *at minimum* the - following attached *or* inherited policy: + .. tab:: Replication Remote User + + The following policy provides permissions for enabling synchronization of + replicated data *into* the cluster. .. code-block:: shell :class: copyable @@ -150,11 +177,13 @@ source and destination clusters: "s3:ListBucketMultipartUploads", "s3:GetBucketLocation", "s3:GetBucketVersioning", - "s3:GetBucketObjectLockConfiguration" + "s3:GetBucketObjectLockConfiguration", + "s3:GetEncryptionConfiguration" ], "Resource": [ - "arn:aws:s3:::DESTINATIONBUCKETNAME" - ] + "arn:aws:s3:::*" + ], + "Sid": "EnableReplicationOnBucket" }, { "Effect": "Allow", @@ -166,23 +195,39 @@ source and destination clusters: "s3:GetObjectVersion", "s3:GetObjectVersionTagging", "s3:PutObject", + "s3:PutObjectRetention", + "s3:PutBucketObjectLockConfiguration", + "s3:PutObjectLegalHold", "s3:DeleteObject", "s3:ReplicateObject", "s3:ReplicateDelete" ], "Resource": [ - "arn:aws:s3:::DESTINATIONBUCKETNAME/*" - ] + "arn:aws:s3:::*" + ], + "Sid": "EnableReplicatingDataIntoBucket" } ] } - Replace ``DESTINATIONBUCKETNAME`` with the name of the target bucket to - which MinIO replicates objects. + - The ``"EnableReplicationOnBucket"`` statement grants permission for + a remote target to retrieve bucket-level configuration for supporting + replication operations on *all* buckets in the MinIO cluster. To + restrict the policy to specific buckets, specify those buckets as an + element in the ``Resource`` array similar to + ``"arn:aws:s3:::bucketName"``. - Use the :mc-cmd:`mc admin policy set` command to associate the policy - to a user on the target MinIO cluster. + - The ``"EnableReplicatingDataIntoBucket"`` statement grants permission + for a remote target to synchronize data into *any* bucket in the MinIO + cluster. To restrict the policy to specific buckets, specify those + buckets as an element in the ``Resource`` array similar to + ``"arn:aws:s3:::bucketName/*"``. + Use the :mc-cmd:`mc admin policy add` to add this policy to the + destination cluster. Use :mc-cmd:`mc admin user add` to create a user + on the destination cluster and :mc-cmd:`mc admin policy set` to associate + the policy to that new user. + MinIO strongly recommends creating users specifically for supporting bucket replication operations. See :mc:`mc admin user` and :mc:`mc admin policy` for more complete @@ -199,6 +244,14 @@ does *not* apply replication rules to existing objects in the bucket. Use :mc:`mc cp` or :mc:`mc mirror` to migrate existing objects to the destination bucket. +For buckets with active write operations during the procedure, any objects +written *before* configuring bucket replication remain unreplicated. + +Consider scheduling a maintenance period during which applications stop +all write operations to the bucket or buckets for which you are configuring +bucket replication. Restart write operations at the completion of the +procedure to ensure consistent object replication. + Replication of Delete Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -237,51 +290,66 @@ MinIO does *not* support replicating client-side encrypted objects Procedure --------- -1) Configure ``mc`` Access to the Source and Destination Cluster +1) Configure User Accounts and Policies for Replication +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This step creates users and policies on both MinIO clusters for +supporting replication operations. You can skip this step if both +clusters already have users with the necessary +:ref:`permissions `. + +The following examples use ``Alpha`` and ``Baker`` as placeholder :mc:`aliases +` for each MinIO cluster. You should replace these values with the +appropriate aliases for the MinIO clusters on which you are configuring bucket +replication. These examples assume that the specified aliases have +the necessary permissions for creating policies and users on both clusters. See +:ref:`minio-users` and :ref:`minio-policy` for more complete documentation on +MinIO users and policies respectively. + +A\) Create Replication Administrator + The following code creates a user and policy for supporting configuring + replication on the ``Alpha`` cluster. Replace the + password ``LongRandomSecretKey`` with a long, random, and secure secret key + as per your organizations best practices for password generation. + + .. code-block:: shell + :class: copyable + + wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationAdminPolicy.json | \ + mc admin policy add Alpha ReplicationAdminPolicy /dev/stdin + mc admin user add Alpha alphaReplicationAdmin LongRandomSecretKey + mc admin policy set Alpha ReplicationAdminPolicy user=alphaReplicationAdmin + +B\) Create Remote Replication User + The following code creates a user and policy for supporting synchronizing + data into the ``Baker`` cluster. Replace the password + ``LongRandomSecretKey`` with a long, random, and secure secret key as per + your organizations best practices for password generation. + + .. code-block:: shell + :class: copyable + + wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationRemoteUserPolicy.json | \ + mc admin policy add Baker ReplicationRemoteUserPolicy /dev/stdin + mc admin user add Baker bakerReplicationRemoteUser LongRandomSecretKey + mc admin policy set Baker ReplicationRemoteUserPolicy user=bakerReplicationRemoteUser + +2) Configure ``mc`` Access to the Source and Destination Cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the :mc-cmd:`mc alias set` command to add an alias for both source and destination MinIO clusters. -.. tabs:: +Use the :mc-cmd:`mc alias set` command to add a replication-specific alias for +both remote clusters: - .. tab: Source Cluster +.. code-block:: shell + :class: copyable - .. code-block:: shell - :class: copyable + mc admin set AlphaReplication HOSTNAME AlphaReplicationAdmin LongRandomSecretKey + mc admin set BakerReplication HOSTNAME BakerReplicationUser LongRandomSecretKey - mc alias set SourceCluster HOSTNAME ACCESSKEY SECRETKEY - - - Replace :mc-cmd:`~mc alias set HOSTNAME` with the hostname or IP address - of a node in the source MinIO cluster. For distributed MinIO clusters - using a load balancer, specify the hostname or IP address of that load - balancer. - - - Replace :mc-cmd:`~mc alias set ACCESSKEY` and - :mc-cmd:`~mc alias set SECRETKEY` with the access and secret key for a - user with the :ref:`required permissions - ` on the source - MinIO cluster. - - .. tab:: DestinationCluster - - .. code-block:: shell - :class: copyable - - mc alias set DestinationCluster HOSTNAME ACCESSKEY SECRETKEY - - - Replace :mc-cmd:`~mc alias set HOSTNAME` with the hostname or IP address - of a node in the destination MinIO cluster. For distributed MinIO - clusters using a load balancer, specify the hostname or IP address of - that load balancer. - - - Replace :mc-cmd:`~mc alias set ACCESSKEY` and - :mc-cmd:`~mc alias set SECRETKEY` with the access and secret key for a - user with the :ref:`required permissions - ` on the - destination MinIO cluster. - -2) Create a Replication Target for the Destination Cluster +3) Create a Replication Target for the Destination Cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the :mc-cmd:`mc admin bucket remote` command to create a replication target @@ -292,35 +360,24 @@ bucket. .. code-block:: shell :class: copyable - mc admin bucket remote add SourceCluster/SOURCEBUCKET \ - https://ACCESSKEY:SECRETKEY@HOSTNAME/DESTINATIONBUCKET + mc admin bucket remote add AlphaReplication/SOURCEBUCKET \ + https://bakerReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/DESTINATIONBUCKET --service "replication" [--sync] -- Replace ``SOURCEBUCKET`` with the name of the bucket on the source cluster. - The name of the bucket *must* match the bucket name specified to the source - user's replication :ref:`policy - `. + - Replace ``SOURCEBUCKET`` with the name of the source bucket on the + ``Alpha`` cluster. -- Replace ``ACCESSKEY`` and ``SECRETKEY`` with the access and secret key for - a user with the :ref:`required permissions - ` on the destination - MinIO cluster. + - Replace ``HOSTNAME`` with the URL of the ``Baker`` cluster. -- Replace the ``HOSTNAME`` with the hostname or IP address of - a node in the MinIO cluster. For distributed MinIO clusters using a - load balancer, specify the hostname or IP address of that load balancer. + - Replace ``DESTINATIONBUCKET`` with the name of the target bucket on the + ``Baker`` cluster. -- Replace the ``DESTINATIONBUCKET`` with the name of the bucket on the - destination cluster. The name of the bucket *must* match the bucket name - specified to the destination user's replication :ref:`policy - `. - -- MinIO defaults to using asynchronous object replication, where MinIO - replicates objects *after* returning the PUT object response. Specify the - :mc-cmd-option:`~mc admin bucket remote add sync` option to enable - synchronous replication, where MinIO attempts to replicate the object - *prior* to returning the PUT object response. + - Specify the :mc-cmd-option:`~mc admin bucket remote add sync` option to + enable synchronous replication. Omit the option to use the default of + asynchronous replication. See the reference documentation for + :mc-cmd-option:`~mc admin bucket remote add sync` for more information + on synchronous vs asynchronous replication. The command returns an ARN similar to the following: @@ -330,7 +387,7 @@ The command returns an ARN similar to the following: Copy the ARN string for use in the next step. -3) Create a New Bucket Replication Rule +4) Create a New Bucket Replication Rule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use the :mc-cmd:`mc replicate add` command to add the new server-side @@ -339,29 +396,35 @@ replication rule to the source MinIO cluster. .. code-block:: shell :class: copyable - mc replicate add SourceCluster/SOURCEBUCKET \ + mc replicate add AlphaReplication/SOURCEBUCKET \ --remote-bucket DESTINATIONBUCKET \ --arn 'arn:minio:replication:::DESTINATIONBUCKET' \ --replicate "delete,delete-marker" -- Replace ``SOURCEBUCKET`` with the name of the bucket on the source cluster. + - Replace ``SOURCEBUCKET`` with the name of the bucket from which Alpha + replicates data. The name *must* match the bucket specified when + creating the remote target in the previous step. -- Replace the ``DESTINATIONBUCKET`` with the name of the bucket on the - destination cluster. + - Replace the ``DESTINATIONBUCKET`` with the name of the ``Baker`` bucket to + which Alpha replicates data. The name *must* match the bucket specified + when creating the remote target in the previous step. -- Replace the ``--arn`` value with the ARN returned in the previous step. - -- Replace ``PRIORITY`` with the integer priority of this replication rule. + - Replace the ``--arn`` value with the ARN returned in the previous step. + Ensure you specify the ARN created on the ``Alpha`` cluster. You can use + :mc-cmd:`mc admin bucket remote ls` to list all remote ARNs configured + on the cluster. + + - The ``--replicate "delete,delete-marker"`` flag enables replicating delete + markers and deletion of object versions. See + :mc-cmd-option:`mc replicate add replicate` for more complete + documentation. Omit these fields to disable replication of delete + operations. -- The ``--replicate "delete,delete-marker"`` flag enables replicating delete - markers and deletion of object versions. See - :mc-cmd-option:`mc replicate add replicate` for more complete documentation. - Omit this field to disable replication of delete operations. Specify any other supported optional arguments for :mc-cmd:`mc replicate add`. -4) Validate the Replication Configuration +5) Validate the Replication Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use :mc-cmd:`mc cp` to copy a new object to the source bucket. @@ -369,14 +432,14 @@ Use :mc-cmd:`mc cp` to copy a new object to the source bucket. .. code-block:: shell :class: copyable - mc cp ~/foo.txt SourceCluster/SOURCEBUCKET + mc cp ~/foo.txt Alpha/SOURCEBUCKET Use :mc-cmd:`mc ls` to verify the object exists on the destination bucket: .. code-block:: shell :class: copyable - mc ls DestinationCluster/DESTINATIONBUCKET + mc ls Baker/DESTINATIONBUCKET If the remote target was configured *without* the :mc-cmd-option:`~mc admin bucket remote add sync` option, the destination diff --git a/source/replication/enable-server-side-two-way-bucket-replication.rst b/source/replication/enable-server-side-two-way-bucket-replication.rst index 40e12f44..9ac39ba2 100644 --- a/source/replication/enable-server-side-two-way-bucket-replication.rst +++ b/source/replication/enable-server-side-two-way-bucket-replication.rst @@ -14,11 +14,13 @@ Enable Two-Way Server-Side Bucket Replication The procedure on this page creates a new bucket replication rule for two-way "active-active" synchronization of objects between MinIO buckets. -.. todo: Diagram +.. image:: /images/active-active-replication.svg + :width: 600px + :alt: Active-Active Replication synchronizes data between two remote clusters. + :align: center -MinIO server-side replication supports active-active replication between -at most *two* MinIO clusters. MinIO does not support arbitrary S3-compatible -services. +MinIO server-side replication supports at most *two* MinIO clusters. Both +clusters *must* run MinIO. - To configure replication between arbitrary S3-compatible services, use :mc-cmd:`mc mirror`. @@ -26,6 +28,13 @@ services. - To configure one-way "active-passive" replication between MinIO clusters, see :ref:`minio-bucket-replication-serverside-oneway`. +MinIO Active-Active replication is designed for synchronizing objects between +two MinIO clusters. MinIO does not support Active-Active replication between +more than two clusters (multi-site). Enterprises looking to implement multi-site +replication should consider leveraging `MinIO SUBNET +`__ support to access the expertise, planning, +and engineering resources required for addressing that use case. + .. seealso:: - Use the :mc-cmd:`mc replicate edit` command to modify an existing @@ -38,13 +47,6 @@ services. - Use the :mc-cmd:`mc replicate rm` command to remove an existing replication rule. -.. todo: Diagram - -.. todo - This procedure specifically enables only two-way replication between the - source and destination buckets. For a procedure on two-way "active-active" - replication, see . - .. _minio-bucket-replication-serverside-twoway-requirements: Requirements @@ -193,6 +195,7 @@ destination clusters to configure and enable replication rules. "s3:GetObjectVersionTagging", "s3:PutObject", "s3:PutObjectRetention", + "s3:PutBucketObjectLockConfiguration", "s3:PutObjectLegalHold", "s3:DeleteObject", "s3:ReplicateObject", @@ -291,8 +294,6 @@ MinIO supports replicating objects with retention settings, such as source and destination bucket *must* have object locking enabled for MinIO to replicate objects with their associated retention settings. - - Procedure --------- @@ -304,14 +305,13 @@ supporting replication operations. You can skip this step if both clusters already have users with the necessary :ref:`permissions `. -The following examples that the :mc:`alias ` for each cluster -provides the necessary permissions for creating policies and users on both -clusters. See :ref:`minio-users` and :ref:`minio-policy` for more complete -documentation on MinIO users and policies respectively. - -Replace ``Alpha`` and ``Baker`` in the following examples with the appropriate -:mc:`aliases ` for the MinIO clusters on which you are configuring -bucket replication. +The following examples use ``Alpha`` and ``Baker`` as placeholder :mc:`aliases +` for each MinIO cluster. You should replace these values with the +appropriate aliases for the MinIO clusters on which you are configuring bucket +replication. These examples assume that the specified aliases have +the necessary permissions for creating policies and users on both clusters. See +:ref:`minio-users` and :ref:`minio-policy` for more complete documentation on +MinIO users and policies respectively. A\) Create Replication Administrators The following code creates policies and users for supporting configuring @@ -332,7 +332,7 @@ A\) Create Replication Administrators mc admin user add Baker bakerReplicationAdmin LongRandomSecretKey mc admin policy set baker ReplicationAdminPolicy user=bakerReplicationAdmin -B\) Create Remote Replication Administrators +B\) Create Remote Replication User The following code creates policies and users for supporting synchronizing data to the ``Alpha`` and ``Baker`` clusters. Replace the password ``LongRandomSecretKey`` with a long, random, and secure secret key as per your diff --git a/source/replication/replication-overview.rst b/source/replication/replication-overview.rst index b546d335..110e1e11 100644 --- a/source/replication/replication-overview.rst +++ b/source/replication/replication-overview.rst @@ -8,7 +8,7 @@ Bucket Replication .. contents:: Table of Contents :local: - :depth: 1 + :depth: 2 MinIO supports server-side and client-side replication of objects between source and destination buckets. MinIO offers both active-passive (one-way) and @@ -54,8 +54,6 @@ replication while adding the following MinIO-only features: - Source and destination bucket names can match, supporting site-to-site use cases such as Splunk or Veeam BC/DR. -- Replicate object locking and retention settings from source to destination. - - Simplified implementation than S3 bucket replication configuration, removing the need to configure settings like AccessControlTranslation, Metrics, and SourceSelectionCriteria. @@ -63,6 +61,111 @@ replication while adding the following MinIO-only features: - Active-Active (Two-Way) replication of objects between source and destination buckets. +.. _minio-replication-process: + +Replication Process +~~~~~~~~~~~~~~~~~~~ + +MinIO uses a replication queuing system with multiple concurrent replication +workers operating on that queue. MinIO continuously works to replicate and +remove objects from the queue while scanning for new unreplicated objects to +add to the queue. + +MinIO sets the ``X-Amz-Replication-Status`` metadata field according to the +replication state of the object: + +.. list-table:: + :header-rows: 1 + :width: 100% + :widths: 30 70 + + * - Replication State + - Description + + * - ``PENDING`` + - The object has not yet been replicated. MinIO applies this state + if the object meets one of the configured replication rules on the + bucket. MinIO continuously scans for ``PENDING`` objects not yet in the + replication queue and adds them to the queue as space is available. + + * - ``COMPLETED`` + - The object has successfully replicated to the remote cluster. + + * - ``FAILED`` + - The object failed to replicate to the remote cluster. + + MinIO continuously scans for ``FAILED`` objects not yet in the + replication queue and adds them to the queue as space is available. + + * - ``REPLICA`` + - The object is itself a replica from a remote source. + + MinIO ignores ``REPLICA`` objects, including any metadata-only changes + to that object. MinIO does not support + AWS `replica modification sync + `__ + at this time. + +The replication process generally has one of the following flows: + +- ``PENDING -> COMPLETED`` +- ``PENDING -> FAILED -> COMPLETED`` + +Synchronous vs Asynchronous Replication +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO supports specifying either asynchronous (default) or synchronous +replication for a given remote target. + +With the default asynchronous replication, MinIO completes the originating +``PUT`` operation *before* placing the object into a :ref:`replication queue +`. The originating client may therefore see a +successful ``PUT`` operation *before* the object is replicated. While +this may result in stale or missing objects on the remote, it mitigates +the risk of slow write operations due to replication load. + +With synchronous replication, MinIO attempts to replicate the object *prior* to +completing the originating ``PUT`` operation. MinIO returns a successful ``PUT`` +operation whether or not the replication attempts succeeds. While this may +result in more reliable synchronization between the source and remote target, +it may also increase the time of each write operation due to replication load. + +You must explicitly enable synchronous replication when configuring the remote +target target using the :mc-cmd:`mc admin bucket remote add` command with the +:mc-cmd-option:`~mc admin bucket remote add sync` flag. + +Replication of Delete Operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +MinIO supports replicating delete operations, where MinIO synchronizes +deleting specific object versions *and* new :s3-docs:`delete markers +`. Delete operation replication uses +the same :ref:`replication process ` as all other +replication operations. + +For delete marker replication, MinIO begins the replication process after +a delete operation creates the delete marker. MinIO uses the +``X-Minio-Replication-DeleteMarker-Status`` metadata field for tracking +delete marker replication status. In +:ref:`active-active ` +replication configurations, MinIO may produce duplicate delete markers if +both clusters concurrently create a delete marker for an object *or* +if one or both clusters were down before the replication event synchronized. + +For replicating the deletion of a specific object version, MinIO marks the +object version as ``PENDING`` until replication completes. Once the remote +target deletes that object version, MinIO deletes the object on the source. +While this process ensures near-synchronized version deletion, it may result +in listing operations returning the object version after the initial +delete operation. MinIO uses the ``X-Minio-Replication-Delete-Status`` for +tracking delete version replication status. + +MinIO requires explicitly enabling versioned deletes and delete marker +replication . Use the :mc-cmd-option:`mc replicate add replicate` field to +specify both or either ``delete`` and ``delete-marker`` to enable versioned +deletes and delete marker replication respectively. To enable both, specify both +strings using a comma separator ``delete,delete-marker``. + .. toctree:: :hidden: :titlesonly: